[Twisted-Python] data dispatch on massive connection counts

Itamar Turner-Trauring itamar at itamarst.org
Mon Nov 14 08:02:32 EST 2011


On Mon, 2011-11-14 at 04:07 -0800, Tobias Oberstein wrote:
> I am testing a WebSocket based pubsub system .. I have 2 questions .. any hints welcome!
> 
> Environment:
>   + FreeBSD 8.2 i386
>   + Python 2.7.2 32-bit
>   + Twisted Trunk
>   + new kqueue() reactor
> 
> The FreeBSD is tuned for massive connection numbers. I can connect 50k WS clients. Fine.
> 
> 
> 1) Massive dispatching
> 
> Essentially, what I'm currently doing is:
> 
> recvs = set([100k instances of protocol.Protocol])
> 
> data = "..."
> for recv in recvs:
>    recv.transport.write(data)
> 
> Now, writing mass-data to _one_ transport should be done using producer/consumer.
> 
> But in my case, the data itself is tiny (<100 octets) and the same for all clients, but the number
> of transports to dispatch that data to is massive.
> 
> The problem is: while above loop is running, other stuff is being delayed.
> 
> What would be the right approach to solve that?

Besides callLater, a higher level construct to do this is
twisted.internet.task.cooperate().
https://blip.tv/pycon-us-videos-2009-2010-2011/pycon-2010-cooperative-multitasking-with-twisted-getting-things-done-concurrently-11-3352182 is a presentation dreid did, which should really be turned into a howto.

> 2) Too many files.
> Is there another limit specifically for files, and/or something tunable in Python/Twisted?

That's definitely not a limit on Python or Twisted. That's an OS limit.
Are you opening lots of files in addition to your sockets?




More information about the Twisted-Python mailing list