[Twisted-Python] Reentrant reactor iteration

[Marcel Keller]

Hi,

I am working on the VIFF project (viff.dk) which uses Twisted. I found 
out that our code is sometimes inefficient because we are generating 
many deferreds (maybe about 10000) in a callback. While doing that, no 
network communication is performed. Therefore, I investigated the 
possibility of adding a function to the reactor which is called every 
iteration and from which the iteration could be called safely. Then, we 
could generate all deferreds in that function and activate the reactor 
from to time. See the attached patch for details.

Some of our code runs twice as fast when using that hack. Are there any 
chances something similar might be included in Twisted? Or does anyone 
have a better solution for the described problem?

Furthermore, I have a question, which is probably related. The 
documentation of IReactorCore says about the iterate() function:

"The reactor must have been started (via the run() method) prior to any 
invocations of this method. It must also be stopped manually after the 
last call to this method (via the stop() method). This method is not 
re-entrant: you must not call it recursively; in particular, you must 
not call it while the reactor is running."

This looks to me as if the reactor needs to be running and not running 
at the same time so that iterate() can be called. Is there an error in 
my reasoning?

Best regards,
Marcel Keller


-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: twisted-hack.patch
Url: http://twistedmatrix.com/pipermail/twisted-python/attachments/20090227/4f7c94aa/attachment.txt