[Twisted-Python] Advise for heavy concurrency

Tristan Seligmann mithrandi at mithrandi.za.net
Sun Dec 2 07:21:25 EST 2007


* Alec Matusis <matusis at yahoo.com> [2007-12-01 23:26:38 -0800]:

> > Because it is not a supported use of the API.  Twisted APIs, unless
> > other-
> > wise documented, are not thread safe and can only be called from the
> > thread
> > in which the reactor is running.
> 
> Then I do not understand when it is safe to use 
> 
> t.i.threads.deferToThread(f, clients) ?

The code in f, or in functions called from f, must not invoke any
functions in the Twisted API other than reactor.callFromThread. The same
goes for a thread created any other way (eg. with callInThread).

> I was thinking of calling this from the main reactor thread, where f is
> 
> def f(clients):
>     for client in clients:
>         client.protocol.transport.write('hello')
> 
> 
> This function would not even have a deferred attached to it. 
> Is the point that it's not thread-safe to have Twisted API calls
> inside the function f?

Yes, that is basically the point; the only API that is thread-safe is
callFromThread; any other Twisted function must be invoked from the
reactor thread.

If this loop is really blocking for too long, you could use something
like the following, which cooperatively distributes the work over
several timeslices; you'll need to check that the default scheduler
performs the work fast enough for your purposes, though.

from twisted.internet.task import coiterate

def f(clients):
    for client in clients:
        client.protocol.transport.write('hello')
        yield None

coiterate(f)
-- 
mithrandi, i Ainil en-Balandor, a faer Ambar
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
Url : http://twistedmatrix.com/pipermail/twisted-python/attachments/20071202/8efaa016/attachment.pgp 


More information about the Twisted-Python mailing list