[Twisted-Python] Throttling a web2 server

Cameron Dale camrdale at gmail.com
Tue Feb 26 21:41:47 EST 2008


On 2/26/08, Jean-Paul Calderone <exarkun at divmod.com> wrote:
> On Tue, 26 Feb 2008 12:12:22 -0800, Cameron Dale <camrdale at gmail.com> wrote:
>  >The trouble I'm having is that the ThrottlingFactory doesn't throttle
>  >the server at all. I've traced the output, and it seems to be due to
>  >the twisted.internet.tcp.Server also calling the pauseProducing() and
>  >resumeProducing() methods of the producer.
>  >
>  >The way the ThrottlingFactory works, is that it schedules periodic
>  >checks of the number of bytes written using reactor.callLater. If too
>  >many bytes have been written, then the producer is paused. However,
>  >these periodic checks always occur when the producer is already paused
>  >by the twisted.internet.tcp.Server, and so have no effect. Later the
>  >producer is resumed by the twisted.internet.tcp.Server, and so the
>  >throttling is non-existent.
>
> Yes.  It would be good if ThrottlingFactory were fixed.  Would you like
>  to fix it? :)

I will have to do something as throttling is needed, but I'm just not
sure how to go about it. Do you have an idea of how it could be fixed?

The ThrottlingFactory methodology would seem to be broken and unusable
due to this behaviour. The only workaround I can think of is to modify
the push producer to keep a tally of the number of pauses received,
and only resume when it gets back to 0. That obviously requires
modifying the underlying producer, and is far from ideal as I believe
the pauseProducing() call is expected to be idempotent.

Or perhaps the ThrottlingFactory should create a dummy producer that
gets data from the real producer, but then throttles it before sending
it out to the twisted.internet.tcp.Server. That is complicated by the
fact that the producer would not be available to be modified by the
factory before it is passed to the Server, so exchanging it for the
dummy producer would also be a problem.

So, are there any other ideas of how throttling can be accomplished
with Twisted? I'm sure someone must have used throttling at some
point.

Thanks,
Cameron




More information about the Twisted-Python mailing list