[Twisted-Python] Question regarding async stuff

Glyph Lefkowitz glyph at twistedmatrix.com
Wed Jul 10 21:51:41 EDT 2002


On Wed, 10 Jul 2002 14:30:38 -0400, Greg Fortune <lists at gregfortune.com> wrote:

> I'm working on a network file server right now and am using Twisted as my
> networking framework.  Overall, it's working really well.  The only thing I
> haven't been able to figure out so far is what is async and what is not.  It
> looks like data transfer is async (self.transport.write(msg)), but the
> functions called in the protocol can block the entire twisted main loop.

Anything that returns a Deferred is asynchronous; everything else is
synchronous.  I think that your use of these terms belies a misunderstanding of
what's going on.

> [...] What is replacing Delayed [...]

IReactorTime.callLater(...)

    http://twistedmatrix.com/documents/TwistedDocs/Twisted-0.19.0rc3/twisted/internet/interfaces_IReactorTime.py.html#callLater

> The reason I ask is that it doesn't appear that a twisted server can handle 
> processing multiple requests at the same time.  It appears that a request 
> must come in and be processed before another request can be serviced.  It 
> probably isn't a big deal for me as the number of requests and amount of 
> processing per request will be low, but it could be that I'm totally 
> misunderstanding how the framework works.

Yeah, I think you're misunderstanding something ;).

Protocol.dataReceived is called only when data is available from a network
connection; therefore, partial requests coming in are partially parsed and
buffered by state machines (Protocol instances).

When a full request has been received, the request can be processed.  If
processing that request requires accessing other asynchronous data that's not
yet available, that's fine too -- just do your transport.write(...) to respond
later on, when a different event arrives.  Some parts of the framework
(twisted.spread, twisted.enterprise) make this extremely explicit, by allowing
the user to return a Deferred when their response is not yet ready.

Twisted can be "processing multiple requests at the same time" in the sense
that while it's waiting on data from the network, it won't be blocked, since
all I/O is asynchronous.  It will be "stopped" while doing literal CPU-bound
"processing" of a request; but while this may seem bad if you look at it
naively, 90% of all request-processing you'll do is incredibly brief, and
managing the resources needed to parallelize that processing is an order of
magnitude (or more, thanks to python's global interpreter lock, mutex
contention, context switching, and other thread nastinesses) more intensive
than just running the requests one after another.

This is before we even start talking about the inherent, dangerous complexity
of thread-based approaches to state management; they're inefficient, and
they're often buggy too.

Even given all that, Twisted does have good support for threads when you really
need them.

    http://twistedmatrix.com/documents/TwistedDocs/Twisted-0.19.0rc3/twisted/internet/interfaces_IReactorThreads.py.html

I hope this answers your questions.  What sort of file server are you writing?

-- 
 |    <`'>    |  Glyph Lefkowitz: Traveling Sorcerer   |
 |   < _/ >   |  Lead Developer,  the Twisted project  |
 |  < ___/ >  |      http://www.twistedmatrix.com      |
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://twistedmatrix.com/pipermail/twisted-python/attachments/20020710/f1f7d2a2/attachment.pgp 


More information about the Twisted-Python mailing list