[Twisted-Python] data dispatch on massive connection counts

[Tobias Oberstein]

I am testing a WebSocket based pubsub system .. I have 2 questions .. any hints welcome!

Environment:
  + FreeBSD 8.2 i386
  + Python 2.7.2 32-bit
  + Twisted Trunk
  + new kqueue() reactor

The FreeBSD is tuned for massive connection numbers. I can connect 50k WS clients. Fine.


1) Massive dispatching

Essentially, what I'm currently doing is:

recvs = set([100k instances of protocol.Protocol])

data = "..."
for recv in recvs:
   recv.transport.write(data)

Now, writing mass-data to _one_ transport should be done using producer/consumer.

But in my case, the data itself is tiny (<100 octets) and the same for all clients, but the number
of transports to dispatch that data to is massive.

The problem is: while above loop is running, other stuff is being delayed.

What would be the right approach to solve that?

One idea is to split the loop into 1k chunks and use reactor.callLater to have the sending
called again until all recvs are served.

Reentry of the reactor via reactor.callLater should make other stuff run in-between, right?

Can I use callLater(0, ..) .. that is no delay at all?

And is this approach recommended anyway?


2) Too many files.

As said, the FreeBSD is tuned for massive connections and I can connect 50k clients.

However, the Twisted application not only contains the WebSockets stuff, but also a
Twisted Web based web server.

Somewhere above 30k connections, I'm beginning to see:


twisted/web/server.py, line 132 in process
...
twisted/python/filepath.py, line 643 in open

<type 'exceptions.IOError'>: [Errno 24] Too many open files

However:

[autobahn at autobahnhub ~/AutobahnHub/service]$ sysctl kern.maxfiles
kern.maxfiles: 204800
[autobahn at autobahnhub ~/AutobahnHub/service]$ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 200000
[autobahn at autobahnhub ~/AutobahnHub/service]$ sysctl kern.ipc.maxsockets
kern.ipc.maxsockets: 204800
[autobahn at autobahnhub ~/AutobahnHub/service]$ ulimit
unlimited

It should go well beyond 50k.

Doing an lsof on the app PID, I can see the 50k connected TCPs and <100 open files.

Why is it denying opening more files?

Is there another limit specifically for files, and/or something tunable in Python/Twisted?