[Twisted-Python] IPushProducer - medium volume streaming

[Rutt, Benjamin]

I've got an IPushProducer written with these 3 methods and it works fine
- i.e. resumeProducing() is called, then eventually pauseProducing is
called() when the producer produces data too fast; when the client
disconnects early, the stopProducing method is called, etc..

  def pauseProducing(self):

  def resumeProducing(self):

  def stopProducing(self):

My problem is, the source of the data (a slow non-relational database)
that I send back to the client from within resumeProducing generates
data kind of slowly.  As in, it is quite rare that pauseProducing is
ever called in my tests.  I feel that the data source generates data too
slowly, and have figured out I will block the entire server during the
resumeProducing operation (for example, during the first call to
resumeProducing, I do not generate any data at all for the first 20
seconds during bootstrapping).

At this point, I am thinking about the following ideas of code changes
to prevent blockage of the server:

1) forking off a thread so that I will not block other clients from
talking to the server.  I will not have very many clients using this
producer, so this would not result in any large numbers of threads which
I know can be a problem.  I know how to spawn Python threads and use
socket APIs, but it is not clear to me how I would get at the socket
descriptor from within the twisted framework (I am using a LineReceiver
subclass, which in this case is spawning the IPushProducer).  Are there
any examples of spawning off a thread from a LineReceiver class and
communicating using blocking calls within said thread?

2) returning from resumeProducing after a few seconds of production,
even though I could produce more, so I do not block the server.  I have
tested this, it works, and does let other clients get in, but I still
feel it is suboptimal, since my data source is so slow, it still blocks
the server during each of those few second intervals.

Any recommendations here?  Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20071026/15c6fead/attachment.htm