[Twisted-Python] Re: Advise for heavy concurrency

Nicola Larosa nico at teknico.net
Sun Dec 2 04:54:13 EST 2007


Atilla wrote:
> When it comes to deferToThread - is that actually going to be of any
> use? Do I get an advantage of running the "send" loop in a seperate
> thread? What if, say, I never had to send more than 10-50 messages, so
> the loop is relatively small - then I'd have to keep spawning threads
> for no reason.

Exactly, no reason. Don't spawn threads, unless you have no other way.


> But - when it comes to large send queues, how can that be chopped in
> an efficient manner? Chop it into equal pieces and schedule them with
> small delay, one after the other?

Yes, and the delay can be very small. Send 50-100 messages at a time, then
run the next batch with a reactor.callLater(0.00000001, ...). That way the
reactor will be able to handle network events with low latency.

Or just use coiterate, as Manlio said:

http://twistedmatrix.com/documents/current/api/twisted.internet.task.html

It works in 10ms batches: that's your latency. It also uses a 10us delay on
the callLater.


> Defer them to another thread?

Don't. Only use threads in Twisted if you have to call external, synchronous
libraries, like database clients.


-- 
Nicola Larosa - http://www.tekNico.net/

Restricting yourself to the intersection of the two versions [Python 2.6
and 3.0] is very painful and limited. We're not introducing backwards
compatibility syntax in 3.0, because that would defeat the purpose
(we've had backwards compatibility forever in 2.x, and the whole point
of 3.0 is to clean up the mess). -- Guido van Rossum, July 2007






More information about the Twisted-Python mailing list