[Twisted-Python] Patch for twisted.protocols.nntp

Jp Calderone exarkun at intarweb.us
Mon Jun 23 12:16:47 EDT 2003

On Mon, Jun 23, 2003 at 12:51:40PM +0200, Matthias Urlichs wrote:
> Hi, Jp Calderone wrote:
> > Hmm, this is the only part of the patch I am unsure about.  The API
> > seems a little too tuned to your use-case.  I think the way to go for
> > this would be to have a Producer passed in and make the NNTP protocol a
> > Consumer for that (in turn acting as a Producer for its transport
> > object).  Would you be willing to make this change? (If you need an
> > example of how this might work, check out smtp.py)
> Hmm. Looking at the SMTP case, it needs refactoring in that the producer
> isn't decoupled from the file object it ends up reading, thus I can't plug
> a different kind of producer in there.

  Yea, it isn't perfect.

> Looking further, there's a slightly-broken data-from-file producer in
> doc/historic/2003/pycon/twisted-internet/twisted-internet.py ...
> Anyway, I need NNTP to be a streaming protocol. With a Deferred, this is
> easy -- each Deferred gets a callback which writes TAKETHIS plus the
> message to the server (and queues the result code recognition), so these
> can complete in any order, which is very nice.

  They can complete in any order because the code responsible for the
Deferred's callback must buffer the message in memory.

> A producer interface would instead have an arbitrary number of producers,
> all of which are active, but as soon as the first one returns anything I
> have to temporarily deactivate all the others. That sounds dangerous --
> what if a producer can't be paused? Then I need to buffer their data, too,
> which is no different from the current case where the article data also
> end up in memory.

  At the very least, the object that is currently a string should become a
file object (This should have minimal impact on your code -- if buffering in
memory is acceptable to you, you only have to build a StringIO object from
your string), and the NNTP protocol class should consume it in a manner
similar to that of smtp.py.  It -does- seem that this is becoming a common
task (smtp.py and pop3.py do it, imap4.py needs to do, and now nntp.py), so
maybe I'll write a utility mixin to help out, and I'll see if I can
generalize it away from just file-like objects too.

  This gives clients at least the opportunity to avoid loading entire
messages into memory, which is what I am looking for.  Am I right in
thinking this leaves you with all the functionality you want?


"Minerals are inexhaustible and will never be depleted. A stream of
investment creates additions to proved reserves, a very large in-ground
inventory, constantly renewed as it is extracted... How much was in the
ground at the start and how much will be left at the end are unknown and
irrelevant."  -- Morry Adelman, World Renowned Economist
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://twistedmatrix.com/pipermail/twisted-python/attachments/20030623/749b3f5b/attachment.pgp 

More information about the Twisted-Python mailing list