[Twisted-Python] Twisted scalability with hundreds of outbound (client) connections

[Itamar Shtull-Trauring]

On Sun, 2004-09-26 at 15:35, Phil Mayers wrote:

> 
> def xmit(self):
>      while True:
>          txqueue = self.txcurrent
>          self.txcurrent = txqueue.next
>          if txqueue:
>              self.write(txqueue.pop(0))
>              break
>      reactor.callLater(0, self.xmit)

This code confuses me. What is the "next" bit for?

> I'm concerned about all those reactor.callLater - since one of the main 
> problems is the UDP socket queue overflowing, every time I xmit I have 
> to get *out* of the protocol code ASAP and back into the select() loop, 
> however one of the problems with the reactors (problems for me at any 
> rate) is that they do pending calls and thread stuff before IO, which 
> IMHO is not quite the right way round.

ABABAB
BABABA

These are indistinguishable once you've done half an iteration, ABABAB
is BABABA slightly timeshifted, it just affects the first and last
iterations.

> I'm also slightly concerned about the number of function calls involved 
> in jumping in and out of the reactor that many times a second (several 
> thousand, if I can get it to go as fast as my previous code) given how 
> expensive they are under Python. It would certainly be quicker to 
> implement this inside the reactor.mainLoop.

Just have a single reactor.callLater(0, f), and f() then calls all the
functions you want done in that iteration.