[Twisted-Python] data dispatch on massive connection counts
Itamar Turner-Trauring
itamar at itamarst.org
Mon Nov 14 08:02:32 EST 2011
On Mon, 2011-11-14 at 04:07 -0800, Tobias Oberstein wrote:
> I am testing a WebSocket based pubsub system .. I have 2 questions .. any hints welcome!
>
> Environment:
> + FreeBSD 8.2 i386
> + Python 2.7.2 32-bit
> + Twisted Trunk
> + new kqueue() reactor
>
> The FreeBSD is tuned for massive connection numbers. I can connect 50k WS clients. Fine.
>
>
> 1) Massive dispatching
>
> Essentially, what I'm currently doing is:
>
> recvs = set([100k instances of protocol.Protocol])
>
> data = "..."
> for recv in recvs:
> recv.transport.write(data)
>
> Now, writing mass-data to _one_ transport should be done using producer/consumer.
>
> But in my case, the data itself is tiny (<100 octets) and the same for all clients, but the number
> of transports to dispatch that data to is massive.
>
> The problem is: while above loop is running, other stuff is being delayed.
>
> What would be the right approach to solve that?
Besides callLater, a higher level construct to do this is
twisted.internet.task.cooperate().
https://blip.tv/pycon-us-videos-2009-2010-2011/pycon-2010-cooperative-multitasking-with-twisted-getting-things-done-concurrently-11-3352182 is a presentation dreid did, which should really be turned into a howto.
> 2) Too many files.
> Is there another limit specifically for files, and/or something tunable in Python/Twisted?
That's definitely not a limit on Python or Twisted. That's an OS limit.
Are you opening lots of files in addition to your sockets?
More information about the Twisted-Python
mailing list