[Twisted-Python] data dispatch on massive connection counts

Tobias Oberstein tobias.oberstein at tavendo.de
Mon Nov 14 09:28:06 EST 2011

> Besides callLater, a higher level construct to do this is
> twisted.internet.task.cooperate().
> https://blip.tv/pycon-us-videos-2009-2010-2011/pycon-2010-cooperative-
> multitasking-with-twisted-getting-things-done-concurrently-11-3352182 is a
> presentation dreid did, which should really be turned into a howto.

Interesting. Need to add that to my toolbox.

In the meantime I've implemented the chunked/callLater approach.

It works.

However, as I currently do it, it breaks message order guarantees .. i.e. it's no
longer guaranteed that each recepient will receive published messages in the
order a _single_ publisher sent them.

Currently I use a set() and pop() from it.

I might change that to fetch receivers from


Is it guaranteed that

callLater(0, fun1)
callLater(0, fun2)

will always result in fun2 being called after fun1?

Because then, and when alway using same chunk size (# of receivers), above
ordering guarantee would hold.

I'd like to avoid having to create a send queue per receiver ..

> > 2) Too many files.
> > Is there another limit specifically for files, and/or something tunable in
> Python/Twisted?
> That's definitely not a limit on Python or Twisted. That's an OS limit.
> Are you opening lots of files in addition to your sockets?


After i.e. 50k clients have connected:

[autobahn at autobahnhub ~/AutobahnHub/service]$ sysctl kern.openfiles
kern.openfiles: 50115

[autobahn at autobahnhub ~/AutobahnHub/service]$ lsof -p 1888 | wc -l

This bugs me .. don't know why it's happening.

More information about the Twisted-Python mailing list