[Twisted-Python] shared list in factory, mutexes?

[Andrew Bennetts]

Gabriel Rossetti wrote:
[...]
> think what I don't get is that with thread network code, each thread is  
> an individual self-contained unit that does whatever processing it needs  
> to do however long it may need to take, and the OS takes care of  
> scheduling each thread, but with twisted, the main thread executes each  
> connection one by one sequentially and thus if one of them needs to to  
> some lengthy processing, it will block all other connections from being  
> processed, so you have to use threads to keep it from blocking, but  
> somehow the active connection should become inactive while the  
> processing thread does it's work, so that another connection can be  
> processed in the meantime. Is that how it works? So twisted's event loop  
> (reactor) does the same work as the OS's scheduler?

In Twisted, code all runs in the main thread, so the way handling multiple
connections works is that control is frequently returned to the reactor.

So if a new connection is received, a protocol instance is made and its
"connectionMade" method called, and as soon as that is done the reactor goes on
to the next thing.  That might be another connectionMade for a new connection,
or it might be delivering a few bytes to a protocol (via calling dataReceived on
it).  So there a lots of events, but the event handlers (i.e. things like
methods on Protocol subclasses) don't block, so they return control very
quickly.

In threaded network code typically you frequently do "sock.recv()" and similar,
to block.  In Twisted you just omit that; your protocol's dataReceived will be
called whenever there's some data.  This frees up the main thread to be doing
other work when there's nothing happening on an individual connection.  So
that's how multiple connections can be handled concurrently in a single thread
in Twisted.

The only rule is that you cannot block in the main thread.  If you want to do
something blocking, find a different way to do it (or if you have to, do it in a
thread and let the main thread continue, but at least for network I/O that's
never necessary in Twisted).

Does that help?

-Andrew.