[Twisted-Python] Deferred documentation.

Jasper St. Pierre jstpierre at mecheye.net
Mon Mar 28 05:52:58 EDT 2011

OK, I'm already making a lot of these changes, I should have a new version
Up Soon (TM).

> Placeholders.
> Placeholders for what, though?

Placeholders for that magic lib, that I think I'm going to stop looking for
and say, "demonstration purposes only"

How do they use it?  Why is it relevant?

I guess I'm stupid or slow, but it took a while for me to realize that
Deferreds were basically a standardized callback mechanism. It's not
really written anywhere on the tin: Deferred was to me a bit of an
unobvious name for what it does, and before recently I've always
associated it tightly to scheduling and the reactor.

No, this is a common problem.  I think it would be great to address the
definition a bit more comprehensively.

Through example?

It does seem to be a bit disconnected from the flow of the regular text.
>  Perhaps it would be better if it were laid out as a paragraph, and then
> each sentence, or clause, were examined more closely, perhaps with an
> accompanying code snippet to clarify it.

As I said, I don't want to explain that snippet. I just want to put it
there, ignore it, and hopefully the user will understand more of it as the
tutorial goes on.

> Right now, it's a placeholder for that magic library that I haven't found
> yet.
> I can see why you might want to do that.  In the narrative flow between
> urllib and Deferreds, there is a missing step; you may need to fake that
> out.  I'm just saying that you should be very clear and say "this is a fake
> example, merely for the purpose of illustration".  I think that Tornado has
> some ugly callback-based stuff, but it would be better to leave this example
> fake than to try to teach users how to use that.

Will do.

> Are there specific changes you find that could make it harder to read
> for newcomers?
> I thought my previous message was a list of those :-(

Well, it was a draft, and I felt that most of my changes were already
improvements over what was there. I didn't expect it to be "harder to read
for newcomers".

> The concept of Deferreds isn't hard at all, once you understand what
> they are. The subtle nuances and bits of glue code that Twisted are
> the things that can trip someone up, and what I'm still learning. A
> small amount of very specific use cases for Deferreds happen in
> real-world code and I'd like to show the support that Twisted has for
> them built-in.
> My goals for this document are:
>  1) A list of guaranteed rules about Deferreds for reference at any time.
>  2) An introduction to those rules in a format that doesn't require
> knowledge of others.
>  3) Showing techniques or tricks that you can play by "exploiting"
> parts of those rules in the context of a contrived problem.
>  4) Showing the built-in support for it.
> My first goal for this document would be a clear, concise explanation of
> what a Deferred is and why you need it.

As I said, when you understand what a Deferred is, it's not hard. The hard
part is *how* to use it. The urllib, fake library, Deferred example is all I
really want to go over about what a deferred is, because it's not hard. I
want to: untie a lot of the code samples from the reactor, explain in terms
of the two other examples that it's fundamentally the same thing, then and
go on to the techniques like errbacks, chaining callbacks, and how
separating the callback from the request allows things like

> This should help clear up my writing style a bit. I think in terms of
> separating abstraction layers; I always try separate a fact or rule
> from logic or a technique that can follow when you can exploit that
> fact (feel free to ask the people about 'evolution' in
> #python-offtopic). I also try to think of the code being very linear
> when it evolves: a new rule is added, you have a problem, you can
> exploit that rule with a specific technique, the technique is
> standardized.
> Example A:
>  PROBLEM: You need to create a Deferred with a known result
>  RULE: Callbacks will continue running after you've called "callback"
> or "errback"
>  TECHNIQUE: You can create a Deferred, call 'callback' on it and
> return it, without any tricky business
>  STANDARDIZED: twisted.internet.defer.success
> Example B:
>  PROBLEM: You need to get the results from multiple Deferreds without
> blocking or too much linearity
>  RULE: More than one Deferred can be created and 'run' at the same time
>  TECHNIQUE: You can add a callback to a Deferred, take the result you
> get and save it in a list or dictionary
>  STANDARDIZED: DeferredList, gatherResults
> These seem more like recipes to me than introductory documentation.  Maybe
> they should be really close together, to try to drive the concept home, but
> it would be good to really get it clear in the reader's mind why they *
> fundamentally* need Deferreds, then to cover all the subtle *different* ways
> you might need them and how you could use them.

But you don't *fundamenally* *need *Deferreds at all. The new document isn't
going to convince them that they're the right thing or wrong thing, just to
show how to get along in their new home in Deferred-ville, USA. For the
recipes, they're going to be reading real-world Twisted code, they better
know what these recipes do.

And if you can think of some subtly different ways to use them other than
the generic "something is happening in the future, I better be prepared for
it", I'll put it in the document and attempt to eat my hat.

> *snip*
> ( Also, this is a real recipe, ableit simplified and it makes really
> easy, delicious chocolate mousse:
> http://articles.latimes.com/2008/feb/13/food/la-fo-watch13recafeb13 )
> This looks like a fantastic example.  It's comprehensible, concrete, not
> too long, and involves a strict metaphor for a real world situation, without
> mixing in any obscure technology.  I would be happy if the entire Deferred
> tutorial were to be structured around it.

Except it really breaks that "runnable examples" thing. As I said before,
recipes are a very convenient match for asynchronous code, but it's not
realistic to try to map 1:1 to the other.

> You could also tweak it to introduce additional concepts.  For example,
> errbacks: "If the butter burns...".

If the butter burns.... what do I do? Turn off the heat, clean the pan, and
restart some more butter? What if I run out of butter? Do I need a
"goToStoreToFetchButter" function? Is that another tutorial I should write?

> Once I *knew* what a Deferred was, the other pieces started snapping into
> place.
> So it sounds like we're in agreement here: the existing document isn't
> clear enough about exactly what a Deferred is, it's described too formally
> and its uses aren't clear enough before we start diving into the technical
> specifics.  Any modification should strive to make it super clear what it is
> and why you use it.

When I started with Twisted, I used Deferreds because "everything else in
Twisted uses it". I didn't really *try* to understand it too much, either.
Most of the time I used Twisted, I tried to hack it all up with
inlineCallbacks and pretend it was urllib, and then complain in #twisted
with a code snippet when it wasn't working.

Use 'localhost' URLs and have the user run a 'twistd web' command line for
> their server; that should be simple enough :).

For static pages, is there anything fundamentally wrong with "
http://twistedmatrix.com/documents/11.1.0/static/pi.txt", if I can do it?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20110328/07b2b17a/attachment-0001.htm 

More information about the Twisted-Python mailing list