[Twisted-Python] Advise for heavy concurrency

Atilla theatilla at gmail.com
Sat Dec 1 18:37:47 EST 2007


I think I can control a lot of the memory overhead, both the one in my
application and the one that comes with the OSs overhead for the TCP
stack and network bufers. I've yet to see how much memory that is
going to take.

When it comes to deferToThread - is that actually going to be of any
use? Do I get an advantage of running the "send" loop in a seperate
thread? What if, say, I never had to send more than 10-50 messages, so
the loop is relatively small - then I'd have to keep spawning threads
for no reason.

But - when it comes to large send queues, how can that be chopped in
an efficient manner? Chop it into equal pieces and schedule them with
small delay, one after the other? Defer them to another thread? Is
there any other solution, a way to chain them into a log callback
squence maybe?




More information about the Twisted-Python mailing list