[Twisted-Python] doPoll timeout problems

[Scott, Barry]

We are seeing a problem with poll being called very often and think that the 
problem is in the timeout calculation.

The code look like this:

    def doPoll(self, timeout):
        """Poll the poller for new events."""
        if timeout is not None:
            timeout = int(timeout * 1000) # convert seconds to milliseconds

What this does is round down the timeout.

We are experimenting with this:

    def doPoll(self, timeout):
        """Poll the poller for new events."""
        if timeout is not None:
            timeout = int(math.ceil(timeout * 1000)) # convert seconds to 
milliseconds

This rounds up the timeout.

With this we see that the poll() is called far less often.

The reason being that if the timeout is 0.0009 the code uses 0ms timeout.
And if your CPU is fast enough it will do many 0ms timeout calls to poll 
before the timer is removed from the _pendingTimedCalls list.

And if the timeout is 0.0019 the time is 1ms and when the _pendingTimedCalls
is check its first element has not expired yet. So keep doing calls with 1ms.

Typically we see that the first element in the _pendingTimedCalls is a call to 
_updateLogDateTime in HTTPFactory. We use a HTTPFactory many times and also 
NevowSite. That causes a callLater for _updateLogDateTime for each factory.

These _updateLogDateTime call seems to be a lot of complexity for no benefit. 
After all time.time() is very fast on Linux, why cache the log time strings? 
Especially when in our app we never write an access log file. 

We are working on stubbing out all that code for our app to bring our 
performance back up compared to old twisted 2.0.

Barry