[Twisted-Python] Reentrant reactor iteration
exarkun at divmod.com
Fri Feb 27 11:27:23 EST 2009
On Fri, 27 Feb 2009 15:26:43 +0100, Marcel Keller <mkeller at cs.au.dk> wrote:
>I am working on the VIFF project (viff.dk) which uses Twisted. I found out
>that our code is sometimes inefficient because we are generating many
>deferreds (maybe about 10000) in a callback. While doing that, no network
>communication is performed. Therefore, I investigated the possibility of
>adding a function to the reactor which is called every iteration and from
>which the iteration could be called safely. Then, we could generate all
>deferreds in that function and activate the reactor from to time. See the
>attached patch for details.
So you're doing a ton of work all at once now and you want to split up that
ton of work into smaller pieces and do it a little at a time?
If that's the case, then you don't need to modify the reactor, you just
need to split up the work your code is going. There are a lot of techniques
for doing this. coiterate and inlineCallbacks are two solutions which are
closest to "cookie cutter" (ie, you have the least flexibility in deciding
how to use them).
>Some of our code runs twice as fast when using that hack. Are there any
>chances something similar might be included in Twisted? Or does anyone have
>a better solution for the described problem?
Give coiterate or inlineCallbacks a try. If you need help applying either
of these to your problem, please ask. :) I can't make any specific
suggestions now because I can only guess at how you're using the reactor
modification you attached.
You have a very long, steep, uphill battle to convince me that adding support
for re-entrant iteration is a good idea.
>Furthermore, I have a question, which is probably related. The documentation
>of IReactorCore says about the iterate() function:
>"The reactor must have been started (via the run() method) prior to any
>invocations of this method. It must also be stopped manually after the last
>call to this method (via the stop() method). This method is not re-entrant:
>you must not call it recursively; in particular, you must not call it while
>the reactor is running."
>This looks to me as if the reactor needs to be running and not running at
>the same time so that iterate() can be called. Is there an error in my
It's very subtle and intentionally written to sound impossible to use (maybe
that is a stupid idea, but that was my intent when I wrote it ;). What
you're missing is that there is a way to make reactor.run() return without
stopping the reactor.
More information about the Twisted-Python