[Twisted-Python] data dispatch on massive connection counts

[Tobias Oberstein]

> Besides callLater, a higher level construct to do this is
> twisted.internet.task.cooperate().
> https://blip.tv/pycon-us-videos-2009-2010-2011/pycon-2010-cooperative-
> multitasking-with-twisted-getting-things-done-concurrently-11-3352182 is a
> presentation dreid did, which should really be turned into a howto.

Interesting. Need to add that to my toolbox.

In the meantime I've implemented the chunked/callLater approach.

It works.

However, as I currently do it, it breaks message order guarantees .. i.e. it's no
longer guaranteed that each recepient will receive published messages in the
order a _single_ publisher sent them.

Currently I use a set() and pop() from it.

I might change that to fetch receivers from

deque(sorted(set))

Is it guaranteed that

callLater(0, fun1)
callLater(0, fun2)

will always result in fun2 being called after fun1?

Because then, and when alway using same chunk size (# of receivers), above
ordering guarantee would hold.

I'd like to avoid having to create a send queue per receiver ..

> 
> > 2) Too many files.
> > Is there another limit specifically for files, and/or something tunable in
> Python/Twisted?
> 
> That's definitely not a limit on Python or Twisted. That's an OS limit.
> Are you opening lots of files in addition to your sockets?

No.

After i.e. 50k clients have connected:

[autobahn at autobahnhub ~/AutobahnHub/service]$ sysctl kern.openfiles
kern.openfiles: 50115

[autobahn at autobahnhub ~/AutobahnHub/service]$ lsof -p 1888 | wc -l
   50075

This bugs me .. don't know why it's happening.