[Twisted-Python] Learning about IPushProducer

Rutt, Benjamin Benjamin.Rutt at gs.com
Mon Mar 12 17:46:54 EDT 2007

    Sorry, your question was big and challenging to approach.

Understood, thanks.  I should have pared it down.

    This loop:
    >	        while not self.paused and self.howmany >
    >	            chunksz = min(maxchunksz, self.howmany -
    >	            self.consumer.write('x' * chunksz)
    >	            self.sent_already += chunksz
    is a bit atypical, I think.  The reason it is eventually stopping
    is that your code is being invoked re-entrantly by the consumer as
    soon as it decides its buffer is full.  I'm not sure the loop is
    /wrong/, but it is a bit surprising.  You don't need a mutex here,
    since it's single threaded, but you do need to be aware that your
    code can be re-entered within a single thread.
    Does that answer your questions?

Yes, thank you, it answers some of them but it raises more :).  I see,
it's self.consumer.write(...) that ends up calling the pause method.  So
that's how it's reentrant.  I knew twisted wasn't multithreaded in this
case.  Makes sense.

I have since made my resumeProducing() code loop for up to 100 (or 500,
or 1000 etc.) iterations and then return.  Thus I'm treating
resumeProducing as if it should "produce a chunk larger than 1 byte but
smaller that the whole dataset" then return.  How is this approach, does
it better match the intention of the producer/consumer system vs. the
"loop forever until paused" scheme I had earlier?

A final question -- is it safe for my code to ever call pauseProducing,
stopProducing, resumeProducing directly?  Or that will mess up the
balance of the twisted universe?  (I'm thinking that the calls to these
3 methods should only originate "from within twisted" in case the whole
producer/consumer system is tracking how many times it calls each one
etc.)  I don't plan on calling them directly, but want to critically
review in an informed way my colleague's code which does this.

Thank you so much for the response!

Benjamin Rutt

More information about the Twisted-Python mailing list