[Twisted-Python] Question about deferreds
glyph at divmod.com
glyph at divmod.com
Tue May 5 15:31:50 EDT 2009
On 01:48 pm, dfarrell at mypublisher.com wrote:
>I consider myself a Twisted "newbie", so this might seem like a simple
>couple of questions, hope you'll indulge me.
I hope a simple couple of answers will suffice. Thank you for your
clearly-worded questions :).
>When I first began experimenting with deferreds I was breaking up tasks
>into sequences of steps something like this.
>When this runs I was under the mistaken impression that at the end of
>each callback (step1, step2, step3, etc) the deferred would yield back
>to the main loop (reactor) and allow other deferreds to run. Now I know
>that the callback functions run sequentially one after another with no
>break. So my first question is this: what is the advantage of this over
>just combining the callback functions into one big function?, something
>d = Deferred()
Deferreds are not about breaking your work up into steps to save time.
They're about breaking it up so that different systems can deal with the
same asynchronous event in their own way.
For example, let's say you're working on a REST client that deals with
invoices. The server gives you data at a particular URL and you want to
convert it into an "Invoice" instance.
Twisted itself isn't going to have that OneBigStep callback that
converts HTTP straight into invoices for you, so you've got something
that gives back a Deferred that fires with a string, getPage. Then
maybe there's an intermediary layer that deals with parsing XML or JSON
or whatever. Then there's a layer that takes the JSON or DOM structure
and converts it into an Invoice. Finally you've got your actual
application code that wants to get the Invoice and *do* something with
And of course your final "do something with that invoice" step might
need data retrieved asynchronously from other places, so it might need
other Deferreds and other callbacks as well.
If you do have step1, step2, and step3 defined entirely in a single
function, then yes, they should be in the same function.
>If step1(), step2() and step3() take a fair bit of time to complete,
>doesn't this effectively block the reactor loop for the entire exection
>of all the steps?
Yep. That's the idea. If you have long-running tasks that need to be
split up into multiple steps, there are other facilities, *not* Deferred
callbacks, that you can use. For example, this:
>My second question is related to this; what is a recommended way to
>create the cooperative state machine I'm thinking of using deferreds?
>How do I get the execution of this to be this:
>I want to have the state machines I create cooperatively 'multi-task',
>yielding control back to the main loop at each state transition.
If you actually wanted to interleave Deferred callbacks like that it
would be tricky. But if you just want interleaved callbacks, use
Cooperator as I mentioned above :).
>Thanks in advance for any help, guidance or references you can provide.
>I really enjoy Twisted, I'm still working on getting the 'Zen' of it.
More information about the Twisted-Python