[Twisted-Python] Performance issue in reactor.callLater

Stefan Behnel behnel_ml at gkec.informatik.tu-darmstadt.de
Wed Sep 8 03:58:05 EDT 2004

Bob Ippolito schrieb:
 >>>>    heapq.py is under the PSF license.  PSF is roughly similar to MIT.
 >>>> Is this going to be okay for inclusion in Twisted?
 >>> It's part of Python, so I don't see why that would be necessary.
 >> This is patched / somehow different version.

Right, it needs to keep track of the array position where a DelayedCall is stored in order to remove them as quickly as possible.

 >>> However, I have profiled various heapq-based schemes for callLater in
 >>> the past, and they always end up losing pretty big in the average case,
 >>> and winning only in perverse cases such as hundreds of callLater(0)s per
 >>> tick.  I saw a little discussion of the performance impact, but I am
 >>> still not entirely convinced that it's a good idea.
 > What is the "average" case anyway?  For what application?  Where are the
 > performance tests coming from?

Yes, I think that's a problem. Now, as far as I see it, the implementation makes the case where there are a lot of interleaved callLater/cancel/reset calls /much/ faster. And in a case where there is only a few calls it shouldn't matter anyway.

Just to repeat myself: what really get's slower is runUntilCurrent, but the gain in callLater should compensate that. So, for the case where each callLater leads to a delayed call, the performance gain should be marginal (my patch already avoids shifting the list in linear time) and in all cases where there are more calls to callLater, reset and cancel (added up) than launched delayed calls, I expect the heap implementation to be faster by orders of magnitude.

 > There is at least one scenario where it makes perfect sense to do a lot
 > of callLater(0.0, ...), when you're doing computation but trying not to
 > block.
 > Though, of course, you would do something like callLater(0.00001, ...)
 > in the current Twisted, because callLater(0.0, ...) pretty much blocks.

What you may gain in that case is that it is no longer necessary to shift the complete list when adding a new call, so there already is a shift from linear time to log(N).

And again: if there is only a few calls to callLater then this is not the place to look for performance bottlenecks anyway. But that is up to the application programmer.


BTW: I went to http://www.twistedmatrix.com/bugs, but I can't find the obvious link saying "file a bug report HERE". So I registered, but I don't seem to get a reply - and I can't login either. Looks like the web server is running with the /old/ version of Twisted... :o)

More information about the Twisted-Python mailing list