[Twisted-Python] Twisted scalability with hundreds of outbound (client) connections

Itamar Shtull-Trauring itamar at itamarst.org
Sun Sep 26 16:25:35 EDT 2004

On Sun, 2004-09-26 at 15:35, Phil Mayers wrote:

> def xmit(self):
>      while True:
>          txqueue = self.txcurrent
>          self.txcurrent = txqueue.next
>          if txqueue:
>              self.write(txqueue.pop(0))
>              break
>      reactor.callLater(0, self.xmit)

This code confuses me. What is the "next" bit for?

> I'm concerned about all those reactor.callLater - since one of the main 
> problems is the UDP socket queue overflowing, every time I xmit I have 
> to get *out* of the protocol code ASAP and back into the select() loop, 
> however one of the problems with the reactors (problems for me at any 
> rate) is that they do pending calls and thread stuff before IO, which 
> IMHO is not quite the right way round.


These are indistinguishable once you've done half an iteration, ABABAB
is BABABA slightly timeshifted, it just affects the first and last

> I'm also slightly concerned about the number of function calls involved 
> in jumping in and out of the reactor that many times a second (several 
> thousand, if I can get it to go as fast as my previous code) given how 
> expensive they are under Python. It would certainly be quicker to 
> implement this inside the reactor.mainLoop.

Just have a single reactor.callLater(0, f), and f() then calls all the
functions you want done in that iteration.

More information about the Twisted-Python mailing list