[Twisted-Python] Performance and select.select

[Jean-Paul Calderone]

On Fri, 04 Jan 2008 03:42:10 +0100, Nitro <nitro at dr-code.org> wrote:
>Hello,
>
>I profiled my application with cProfiler. Below is an excerpt from the 
>resulting report:
>
>Fri Jan 04 01:07:39 2008    profiling_data_cProfile
>
>          15123197 function calls (14579262 primitive calls) in 228.517 CPU 
>seconds
>
>    Ordered by: internal time
>    List reduced from 2933 to 50 due to restriction <50>
>
>    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>    138305  135.419    0.001  135.817    0.001 {select.select}
>     26168   13.159    0.001   14.044    0.001 
>Sources\Server\Plugins\RKSimulation.py:218(UpdatePhysics)
>
>As you can see select.select takes up ~136 of ~229 seconds. This is the 
>most expensive function. The row below shows the next expensive function 
>which is one of my own. Now I am wondering whether
>
>a) the select calls are really taking up that much time (more than 50%!)
>b) if they are taking up that much time I'd like to find out how to reduce 
>their execution time (either by minimizing the number it gets called or  the 
>time it takes to execute etc)
>
>My application doesn't invoke select.select itself so it is most likely 
>called within twisted. I am on Windows and using the default reactor  ("from 
>twisted.internet import reactor").

Twisted does use select by default on Windows.  One thing you might try is
one of the other reactors available on Windows; there are two,
win32eventreactor and iocpreactor.  You can read about how to do this here:

http://twistedmatrix.com/projects/core/documentation/howto/choosing-reactor.html

Depending on what your application does, iocpreactor may not be complete
enough to actually run your code.  It lacks some features provided by the
default reactor.

Whether either of these reactors will actually improve the performance of
your application, I don't know.  You pointed out that a lot of time is
spent in select, but this doesn't mean that spending less time inside the
event notification function will make your overall application faster.

Also, you are right to be suspicious of these profile results.  Python has
a very spotty history when it comes to profilers.  They are often inaccurate
or completely wrong.  cProfile is supposed to be better than the previous
attempts, but I haven't personally used it extensively enough (nor at all on
Windows) to say whether this is true.

Some things to consider are:

  * How many simultaneous connections does your application have?  select
    usually scales linearly with this number.  I'm not sure exactly what it
    does on Windows, though.

  * How often do connections have events?  Is data passing through them
    constantly or are they mostly idle?  More events means more trips into
    select, which incurs setup costs more.  Fewer events means select should
    be spending more of its time idling, waiting for events.

  * How often do timed events go off?  These also cause more trips into
    select, since each one causes select to return when it is time to run.
    Do you use reactor.callLater() with a 0 timeout a lot?

Hope this helps,

Jean-Paul