[Twisted-Python] Re: To thread or not to thread...

Jonathan Lange jml at divmod.com
Tue Aug 3 01:29:48 EDT 2004


Britt Green wrote:
> I've been reading some of the online documentation for Twisted, and
> I'm slowly starting to grasp different things. However, I could use
> some help with one point. Suppose I was coding up a chat server, or
> perhaps a MUD. Would I need to use something threaded, as opposed to
> just using reactor.run()? Or would the latter be sufficient?
> 

This seems to be a common stumbling point for newcomers to Twisted.

With Twisted, you don't need threads. You don't want threads. Threads 
are actually harmful: http://www.kuro5hin.org/story/2002/11/18/22112/860

But this doesn't help you write a chat server. How can you do that 
without threads? The answer falls into two parts: how Twisted does it 
and how you use Twisted to do it.

1. How Twisted Does It
When you run a Twisted server, Twisted calls the socket functions 
asynchronously. So, rather than calling a listen function which sits 
there waiting until someone connects, it calls a listen function which 
returns immediately. Then, in the <b>reactor</b> loop, it polls things. 
When something connects _Twisted calls your code_ which it expects to 
_return immediately_. The entire process will wait for your code to return.

So, this leaves a bit of a problem: How do I do stuff that takes time?
The answer is too complicated to go into now. The important bit is that 
your function doesn't compute a value and return it. It returns a 
Deferred -- a promise to return later.


2. How You Get Twisted to do it.

---------------------------------------
port = 8007

class MyProtocol(Protocol):
   def connectionMade(self):
     """someone has connected. do something"""
   def dataReceived(self, data):
     """do stuff with data"""

factory = ServerFactory()
factory.protocol = MyProtocol

reactor.listenTCP(port, factory)
reactor.run()
---------------------------------------

Notice how you don't really call any of your own code?
The code is essentially preparing an object that can be registered with 
Twisted as a server factory.

Every time someone connects to port 8007 on that host, Twisted will 
instantiate an instance of MyProtocol and call connectionMade on it.

Every time someone sends data, Twisted will call dataReceived.

The reactor calls everything. If your connectionMade implementation 
takes ages to return, then the process is in _your_ code, not the 
reactor's. This means the reactor can't listen for data and can't call 
your dataReceived method. All this really means is that your methods 
should return quickly.

Really, that's all you need to know. Your methods should return quickly.
As long as they do that, you don't really need to worry about threads 
because Twisted will take care of concurrency for you.

Recommended Reading:
"Asynchronous Programming" - 
http://www.twistedmatrix.com/documents/howto/async

"Writing Servers" -
http://www.twistedmatrix.com/documents/howto/servers

"Using Deferreds" -
http://www.twistedmatrix.com/documents/howto/defer

"Writing Clients" -
http://www.twistedmatrix.com/documents/howto/clients

"Threads Considered Harmful"
http://www.kuro5hin.org/story/2002/11/18/22112/860


Recommended Actions:
* Try writing a _simple_ client/server app in Twisted. It might even be 
worth manually typing out the "quote of the day" examples in the howtos, 
to force your brain to only deal with each concept as it comes, rather 
than trying to grasp the whole in all it's detail.

* Remember that asynchronous programming is just another style of 
programming.

* Forget about threads. You almost never need them. Especially not in 
Twisted.

cheers,
jml





More information about the Twisted-Python mailing list