[Twisted-Python] way of dealing with returning a deferred from a method/function

Brian Warner warner at lothar.com
Wed Dec 3 23:42:23 EST 2003

> On Dec 3, 2003, at 8:48 PM, Jonathan Simms wrote:
>> This may seem obvious to everyone, but I had an insight into the use of
>> deferreds today. It would always confuse me, I've got a method that's
>> going to take quite a while to run.

It occurred to me that the wording here ("quite a while to run") is
significant, and perhaps we should take a closer look at the Deferred
documentation because it might be fueling some of the confusion surrounding
when and where Deferreds will help you.

 bad language:
   The function takes a long time to run.

 good language:
   The function will not be complete for a long time.
   The operation will not finish right away.

Functions which take a long time to run, like "for i in range(10**10):", are
going to take many seconds of continuous CPU time to finish. To do something
else while it is running means time-slicing (run it in a thread), or
task-switching (break it into pieces and chain them with callLater so that
other events can be processed in between the chunks), or a different form of
task-switching (run the operation in a separate OS process and watch for the
child to exit). These are all fundamentally the same technique, the only
difference is who gets to decide how the CPU cycles are allocated and when
the job-switches take place.

A function which may not be complete for a while, like "waitForUserInput()",
or "waitForWebResponse()", use a little CPU to set up the operation and then
sits around doing nothing until somebody else does something (at least they
do when implemented in the old-style synchronous single-tasking blocking-IO
way). They aren't burning CPU in the meantime: they are waiting for someone
else to burn CPU (the web server, the human thinking about which button they
want to push), or for packets to reach their destination and back, or
whatever. Because the operation is not consuming CPU time, and because you
aren't really sure exactly when the operation is going to be complete, it
feels silly to sit around and do nothing for all that time. This is where
Deferreds are useful. They provide a standard idiom for "tell me when
something happens" callbacks.

The important distinction to make is that, in the Deferred case, the
operation is completed by some *external* event (external to the function
that creates the Deferred, that is). It could be network activity (in which
case the event is when the select loop sees the readable socket and calls
dataReceived()), it could be a callLater timer expiring, it could be some
third piece of code responding to an externality and running your callback,
allowing your Deferred to complete. If the completing activity were performed
by the same code that generated (and then returned) the Deferred, you might
as well just do the entire operation synchronously.

The term "blocking" is frequently used in this context. A function that has
to sort a million numbers will take a long time to run, but it is not usually
described as blocking, because there is no way it could return sooner than
that and still get its job done. A function that sends a packet and
busy-waits until another packet comes back *is* said to be blocking.

It's all a question of terminology :)


More information about the Twisted-Python mailing list