[Twisted-Python] Deferred documentation.

[Glyph Lefkowitz]

On Mar 25, 2011, at 7:28 AM, Jasper St. Pierre wrote:

> Big wall of text incoming. If you're going to read any part of this
> email, search for *IMPORTANT* and read that part.
> 
> Right now I'm stuck at creating a simple example for "deferred dependencies".
> 
> On Tue, Mar 22, 2011 at 10:03 PM, Glyph Lefkowitz
> <glyph at twistedmatrix.com> wrote:
>> On Mar 21, 2011, at 9:30 PM, Jasper St. Pierre wrote:
>> 
>>> On IRC, exarkun, glyph, spiv and idnar encouraged me to do a bit of
>>> work on the Defer documentation.
>> 
>> Yay!  This documentation could definitely use some work.
>> 
>>> I kept get confused between the things like returning a Deferred from a
>>> callback and chainDeferred, which I found out wasn't that useful:
>> 
>> Yeah, chainDeferred is not a great method.  Now that Deferreds are
>> non-recursive, I think it's purely worse than inserting an additional
>> Deferred as a result from a callback.
>> 
>> My eventual goal is to reduce the number of documentation about defer
>> down to a near-impossible two documents. I'm hoping to merge some of
>> the good stuff of the other thousands of documents.
>> 
>> That would be absolutely great.
>> 
>>> Thoughts so far?
>> 
>> While I applaud your intent, these drafts look quite rough.  The random
>> interjections and asides in
>> <http://magcius.mecheye.net/twisted/DeferHowTo-Fixup.html> seem distracting
>> and confusing to me.  Trying to put myself in the mind of a newcomer, I find
>> myself asking many questions which are irrelevant to what I'm trying to
>> learn:
>> 
>> What's "async"?  Why is it hard?  (The original document mentions
>> asynchronous stuff, but in the context of full english sentences.)
> 
> Should I mention blocking with something like urllib, then showcase
> another example library that uses regular callbacks?

That sounds like a good idea, actually.  Start by looking at it blocking, explain the problem with that; then demonstrate the one-callback-when-you're-done approach and explain some issues with that, and then move on to Deferreds.

>> Where are we going shopping?  What does shopping have to do with this?
> 
> Pop culture reference. Killed.

To be clear: I did get that reference, actually, and in another context it would be pretty funny :).  Just not in these docs that are supposed to be universally accessible.

>> What's gevent?  Does this have something to do with Twisted?
>> What's node.js?  This looks like Javascript, what does it have to do with
>> Twisted, which I thought was in Python?
> 
> Placeholders.

Placeholders for what, though?

>> (Now I've gotten distracted and I'm reading about gevent and node.js rather
>> than making my Twisted application work and completing the Deferred
>> tutorial.  Epic fail.  But, if I were to continue...)
>> 
>> What's "this pattern"?  Functions?  Don't lots of programs use functions?
> 
> I guess this is a bit obvious when you have first-class-functions as a
> language feature, but it's still a "pattern."

My point was that the antecedents get a little ambiguous by that point in the text.

>> How do they use it?  Why is it relevant?
> 
> I guess I'm stupid or slow, but it took a while for me to realize that
> Deferreds were basically a standardized callback mechanism. It's not
> really written anywhere on the tin: Deferred was to me a bit of an
> unobvious name for what it does, and before recently I've always
> associated it tightly to scheduling and the reactor.

No, this is a common problem.  I think it would be great to address the definition a bit more comprehensively.

>> Why is the first explanation of what a Deferred is referred to as "technical
>> mumbo-jumbo"? Is this really complicated?  If I am not super good at
>> programming already, should I not be reading this?
> 
> It was prefixed with "The abstract" before. I put it back to "The
> abstract". To me, it seems like it's written in a way that makes sense
> only if you understand what a Deferred is, but it was useful, so I
> didn't rip it out. I see it as a paragraph that will make more sense
> as you're reading the article, and once you go back and understand it,
> there's that happy "snap" feeling as you get the concept.

It does seem to be a bit disconnected from the flow of the regular text.  Perhaps it would be better if it were laid out as a paragraph, and then each sentence, or clause, were examined more closely, perhaps with an accompanying code snippet to clarify it.

>> What's an "operation"?  Does that mean 'function' or 'method' or some other
>> special thing?  It says "most operations in Twisted return a Deferred"; but
>> I've called lots of functions in Twisted which returned other objects before
>> reading this tutorial, or returned None.  Were those actually Deferreds?
> 
> Yeah, I need to reword that. How about, "because Deferreds are a core
> part of Twisted, a lot of functions return them"? No... that's not
> good either.

That's closer, but it should be something more specific than "a lot".
> 
>> Instead, the "fixup" changes the first example to rely on a fake library,
>> which will raise exceptions if I try to run it, but doesn't actually explain
>> that 'magiclib' isn't real.  This isn't a huge problem in and of itself (it
>> is trying to demonstrate the "wrong" way to do things, after all) but it
>> sets up the expectation that the rest of the examples are fake, too, and I
>> shouldn't bother to run them.
> 
> Right now, it's a placeholder for that magic library that I haven't found yet.

I can see why you might want to do that.  In the narrative flow between urllib and Deferreds, there is a missing step; you may need to fake that out.  I'm just saying that you should be very clear and say "this is a fake example, merely for the purpose of illustration".  I think that Tornado has some ugly callback-based stuff, but it would be better to leave this example fake than to try to teach users how to use that.

>> I think the original document has plenty of issues, but these changes look
>> like they've been written for people who already mostly understand
>> Deferreds, but are having trouble catching some of the nuances, and need
>> humor to diffuse their frustration and more examples to illustrate different
>> usage patterns, rather than a fundamentally clearer or better explanation
>> than what was offered before.
> 
> *IMPORTANT*
> 
> Are there specific changes you find that could make it harder to read
> for newcomers?

I thought my previous message was a list of those :-(

> The concept of Deferreds isn't hard at all, once you understand what
> they are. The subtle nuances and bits of glue code that Twisted are
> the things that can trip someone up, and what I'm still learning. A
> small amount of very specific use cases for Deferreds happen in
> real-world code and I'd like to show the support that Twisted has for
> them built-in.
> 
> My goals for this document are:
> 
>  1) A list of guaranteed rules about Deferreds for reference at any time.
>  2) An introduction to those rules in a format that doesn't require
> knowledge of others.
>  3) Showing techniques or tricks that you can play by "exploiting"
> parts of those rules in the context of a contrived problem.
>  4) Showing the built-in support for it.

My first goal for this document would be a clear, concise explanation of what a Deferred is and why you need it.

> This should help clear up my writing style a bit. I think in terms of
> separating abstraction layers; I always try separate a fact or rule
> from logic or a technique that can follow when you can exploit that
> fact (feel free to ask the people about 'evolution' in
> #python-offtopic). I also try to think of the code being very linear
> when it evolves: a new rule is added, you have a problem, you can
> exploit that rule with a specific technique, the technique is
> standardized.
> 
> Example A:
>  PROBLEM: You need to create a Deferred with a known result
>  RULE: Callbacks will continue running after you've called "callback"
> or "errback"
>  TECHNIQUE: You can create a Deferred, call 'callback' on it and
> return it, without any tricky business
>  STANDARDIZED: twisted.internet.defer.success
> 
> Example B:
>  PROBLEM: You need to get the results from multiple Deferreds without
> blocking or too much linearity
>  RULE: More than one Deferred can be created and 'run' at the same time
>  TECHNIQUE: You can add a callback to a Deferred, take the result you
> get and save it in a list or dictionary
>  STANDARDIZED: DeferredList, gatherResults

These seem more like recipes to me than introductory documentation.  Maybe they should be really close together, to try to drive the concept home, but it would be good to really get it clear in the reader's mind why they fundamentally need Deferreds, then to cover all the subtle different ways you might need them and how you could use them.

> The hardest part is creating simple, short, runnable code that
> introduces: a problem that doesn't seem silly, a rule that guides
> toward the solution, the technique that uses the rule to solve it. It
> was much easier when I could contrive examples of a network-enabled
> kitchen: recipes map pretty well to code, especially Twisted async:
> 
>  1) Melt butter in a saucepan. When the butter is finished melting,
> put cocoa powder in.
>  2) Meanwhile, beat egg whites and sugar, and cream of tartar.
>  3) When both are done, put the chocolate mix in a Cuisinart, and
> fold in the egg whites.
> 
> Here you have dependencies (butter needs to be melted before cocoa
> powder), multi-tasking (you don't want to wait for the butter while
> beating the eggs), and a way of knowing when things are done (so you
> can fold them into the cuisinart).
> 
> ( Also, this is a real recipe, ableit simplified and it makes really
> easy, delicious chocolate mousse:
> http://articles.latimes.com/2008/feb/13/food/la-fo-watch13recafeb13 )

This looks like a fantastic example.  It's comprehensible, concrete, not too long, and involves a strict metaphor for a real world situation, without mixing in any obscure technology.  I would be happy if the entire Deferred tutorial were to be structured around it.

You could also tweak it to introduce additional concepts.  For example, errbacks: "If the butter burns...".

>> That makes sense, since based on what you've
>> said on #twisted, that's basically the position you find yourself in :).
>>  This document is supposed to be a tutorial though, explaining how to use
>> Deferreds to users who really have no idea what they are (despite its
>> unfortunate name, "Deferred Reference" - that should probably be changed).
>> One thing I think is very good about this attempted rework, though, is the
>> explanation of the motivation for having Deferreds at all, before explaining
>> how they work.  In the current documentation, it's very unclear why we have
>> such an object in the first place, or what the alternatives to it are.
> 
> Once I *knew* what a Deferred was, the other pieces started snapping into place.

So it sounds like we're in agreement here: the existing document isn't clear enough about exactly what a Deferred is, it's described too formally and its uses aren't clear enough before we start diving into the technical specifics.  Any modification should strive to make it super clear what it is and why you use it.

>> However, the example presented makes it seem as though you really don't
>> need Deferreds, because the only problem with the single-callback approach
>> is handling errors. Another major motivation is the ability to return a
>> Deferred through a system with several layers, changing the return value at
>> each layer by post-processing it a bit.  (One possible example: a REST API
>> that wants to deal with objects, and goes via a translation of [bytes from
>> HTTP]->[JSON dicts/lists from parsing those bytes]->[domain-specific objects
>> by converting JSON objects according to the particular API's spec].)
> 
> I never really thought about it before. I just realized right now,
> writing this email, that things like DeferredList aren't cleanly
> possible if the callback is tied to the request.
> 
> Additionally, is the showcase of this in the fixup with "Multiple
> Callbacks" I did fine? The SQL to HTML example that was there before
> seemed a bit contrived, and I wanted to showcase it in a runnable
> snippet that required Twisted. We can't have them install a SQL
> server, so I used xml.minidom instead of lxml, even though know it's
> complete crap. When I'm done, I should replace the www.example.com
> URLs with files hosted on the Twisted doc site. Is there a way to
> point to generate a URL like that with Lore?

Use 'localhost' URLs and have the user run a 'twistd web' command line for their server; that should be simple enough :).

>> However, I think the need would be better illustrated with examples that can
>> actually be run than with fake examples where we assume that the user knows
>> how something like gevent works.  (Also: gevent doesn't actually work this
>> way, for fetching web pages at least, so your example is wrong.  See
>> <http://www.gevent.org/intro.html#monkey-patching>.)
> 
> I've replaced them with "Library A" and "Library B" placeholders. :)

I'm still not really sure you need to talk about other libraries at all, especially not this early in the document.

>> It's pretty easy to write a fake implementation of 'fetchWebPageAsync' which
>> squirrels away the callback somewhere that the example can call it later,
>> and explain that with some handwaving where we say "and pretend that was
>> actually some networking code fetching it".  For that matter, the reactor is
>> introduced too early in the existing docs; we should demonstrate calling the
>> callback synchronously, and then only later introduce a callLater.
> 
> I ripped out all the reactor code in the starting sections of the
> fixup on purpose, and showcased it with what I think is a clear
> example. When introducing the techniques afterwards, I'm going to
> gently ramp up the reactor code to be a bit more real-world.
> 
>> Anyway I hope this wall of text did not discourage you - I just think you
>> need some clearer goals for improving specific aspects of the documentation,
>> and you should write those down first before trying to actually address them
>> with more docs.
> 
> This email did exactly that. Thanks so much!
> 
> P.S. I still have a *lot* to learn. Some tutoring on the subtleties of
> LoopingCall and coiterator/cooperator would be nice. I'm not going to
> even bother to try to explain how inlineCallbacks works to my brain
> right now. I've said enough erroneous fact in IRC. I'm going to need a
> lot more help in the future.

Great.  I look forward to it :).

>> Thanks for your time,
> 
> I appreciate your time a lot more. You have things to do. I don't.

I am a pretty busy guy, but Twisted is open source and community driven: you've got just as many bugs to fix in it as I do ;-).

Please feel free to snip heavily in any replies; anywhere that you feel we've reached an agreement doesn't need more quoting.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20110327/cd03bf3a/attachment-0001.htm