[Twisted-Python] Deferred documentation.

[Jasper St. Pierre]

Big wall of text incoming. If you're going to read any part of this
email, search for *IMPORTANT* and read that part.

Right now I'm stuck at creating a simple example for "deferred dependencies".

On Tue, Mar 22, 2011 at 10:03 PM, Glyph Lefkowitz
<glyph at twistedmatrix.com> wrote:
> On Mar 21, 2011, at 9:30 PM, Jasper St. Pierre wrote:
>
>> On IRC, exarkun, glyph, spiv and idnar encouraged me to do a bit of
>> work on the Defer documentation.
>
> Yay!  This documentation could definitely use some work.
>
>> I kept get confused between the things like returning a Deferred from a
>> callback and chainDeferred, which I found out wasn't that useful:
>
> Yeah, chainDeferred is not a great method.  Now that Deferreds are
> non-recursive, I think it's purely worse than inserting an additional
> Deferred as a result from a callback.
>
> My eventual goal is to reduce the number of documentation about defer
> down to a near-impossible two documents. I'm hoping to merge some of
> the good stuff of the other thousands of documents.
>
> That would be absolutely great.
>
>> Thoughts so far?
>
> While I applaud your intent, these drafts look quite rough.  The random
> interjections and asides in
> <http://magcius.mecheye.net/twisted/DeferHowTo-Fixup.html> seem distracting
> and confusing to me.  Trying to put myself in the mind of a newcomer, I find
> myself asking many questions which are irrelevant to what I'm trying to
> learn:
>
> What's "async"?  Why is it hard?  (The original document mentions
> asynchronous stuff, but in the context of full english sentences.)

Should I mention blocking with something like urllib, then showcase
another example library that uses regular callbacks?

> Where are we going shopping?  What does shopping have to do with this?

Pop culture reference. Killed.

> What's gevent?  Does this have something to do with Twisted?
> What's node.js?  This looks like Javascript, what does it have to do with
> Twisted, which I thought was in Python?

Placeholders.

> (Now I've gotten distracted and I'm reading about gevent and node.js rather
> than making my Twisted application work and completing the Deferred
> tutorial.  Epic fail.  But, if I were to continue...)
>
> What's "this pattern"?  Functions?  Don't lots of programs use functions?

I guess this is a bit obvious when you have first-class-functions as a
language feature, but it's still a "pattern."

> How do they use it?  Why is it relevant?

I guess I'm stupid or slow, but it took a while for me to realize that
Deferreds were basically a standardized callback mechanism. It's not
really written anywhere on the tin: Deferred was to me a bit of an
unobvious name for what it does, and before recently I've always
associated it tightly to scheduling and the reactor.

> Why is Twisted's right hand blue?  (Forget about being a beginner: I
> honestly don't even get this reference.  Googling seems to suggest it has
> something to do with symptoms of heart disease and doesn't seem funny or
> relevant at all.)

Another pop culture reference. Baleeted.

> Why is the first explanation of what a Deferred is referred to as "technical
> mumbo-jumbo"? Is this really complicated?  If I am not super good at
> programming already, should I not be reading this?

It was prefixed with "The abstract" before. I put it back to "The
abstract". To me, it seems like it's written in a way that makes sense
only if you understand what a Deferred is, but it was useful, so I
didn't rip it out. I see it as a paragraph that will make more sense
as you're reading the article, and once you go back and understand it,
there's that happy "snap" feeling as you get the concept.

> What's an "operation"?  Does that mean 'function' or 'method' or some other
> special thing?  It says "most operations in Twisted return a Deferred"; but
> I've called lots of functions in Twisted which returned other objects before
> reading this tutorial, or returned None.  Were those actually Deferreds?

Yeah, I need to reword that. How about, "because Deferreds are a core
part of Twisted, a lot of functions return them"? No... that's not
good either.

> Why do I "not know where this Deferred has been"?  Do Deferreds get dirty or
> broken somehow when I add multiple callbacks?  Should I avoid that?

Again, I fail at humor. Removed.

> The Python examples in the current Deferred Reference are mostly runnable.
>  The ones that aren't, should be.  The documentation should stress that you
> can run these examples simply, and encourage the reader to download and
> experiment with them, and modify them to see what happens when they do
> things in a different order.
> Instead, the "fixup" changes the first example to rely on a fake library,
> which will raise exceptions if I try to run it, but doesn't actually explain
> that 'magiclib' isn't real.  This isn't a huge problem in and of itself (it
> is trying to demonstrate the "wrong" way to do things, after all) but it
> sets up the expectation that the rest of the examples are fake, too, and I
> shouldn't bother to run them.

Right now, it's a placeholder for that magic library that I haven't found yet.

> I think the original document has plenty of issues, but these changes look
> like they've been written for people who already mostly understand
> Deferreds, but are having trouble catching some of the nuances, and need
> humor to diffuse their frustration and more examples to illustrate different
> usage patterns, rather than a fundamentally clearer or better explanation
> than what was offered before.

*IMPORTANT*

Are there specific changes you find that could make it harder to read
for newcomers?

The concept of Deferreds isn't hard at all, once you understand what
they are. The subtle nuances and bits of glue code that Twisted are
the things that can trip someone up, and what I'm still learning. A
small amount of very specific use cases for Deferreds happen in
real-world code and I'd like to show the support that Twisted has for
them built-in.

My goals for this document are:

  1) A list of guaranteed rules about Deferreds for reference at any time.
  2) An introduction to those rules in a format that doesn't require
knowledge of others.
  3) Showing techniques or tricks that you can play by "exploiting"
parts of those rules in the context of a contrived problem.
  4) Showing the built-in support for it.

This should help clear up my writing style a bit. I think in terms of
separating abstraction layers; I always try separate a fact or rule
from logic or a technique that can follow when you can exploit that
fact (feel free to ask the people about 'evolution' in
#python-offtopic). I also try to think of the code being very linear
when it evolves: a new rule is added, you have a problem, you can
exploit that rule with a specific technique, the technique is
standardized.

Example A:
  PROBLEM: You need to create a Deferred with a known result
  RULE: Callbacks will continue running after you've called "callback"
or "errback"
  TECHNIQUE: You can create a Deferred, call 'callback' on it and
return it, without any tricky business
  STANDARDIZED: twisted.internet.defer.success

Example B:
  PROBLEM: You need to get the results from multiple Deferreds without
blocking or too much linearity
  RULE: More than one Deferred can be created and 'run' at the same time
  TECHNIQUE: You can add a callback to a Deferred, take the result you
get and save it in a list or dictionary
  STANDARDIZED: DeferredList, gatherResults

The hardest part is creating simple, short, runnable code that
introduces: a problem that doesn't seem silly, a rule that guides
toward the solution, the technique that uses the rule to solve it. It
was much easier when I could contrive examples of a network-enabled
kitchen: recipes map pretty well to code, especially Twisted async:

  1) Melt butter in a saucepan. When the butter is finished melting,
put cocoa powder in.
  2) Meanwhile, beat egg whites and sugar, and cream of tartar.
  3) When both are done, put the chocolate mix in a Cuisinart, and
fold in the egg whites.

Here you have dependencies (butter needs to be melted before cocoa
powder), multi-tasking (you don't want to wait for the butter while
beating the eggs), and a way of knowing when things are done (so you
can fold them into the cuisinart).

( Also, this is a real recipe, ableit simplified and it makes really
easy, delicious chocolate mousse:
http://articles.latimes.com/2008/feb/13/food/la-fo-watch13recafeb13 )

> That makes sense, since based on what you've
> said on #twisted, that's basically the position you find yourself in :).
>  This document is supposed to be a tutorial though, explaining how to use
> Deferreds to users who really have no idea what they are (despite its
> unfortunate name, "Deferred Reference" - that should probably be changed).
> One thing I think is very good about this attempted rework, though, is the
> explanation of the motivation for having Deferreds at all, before explaining
> how they work.  In the current documentation, it's very unclear why we have
> such an object in the first place, or what the alternatives to it are.

Once I *knew* what a Deferred was, the other pieces started snapping into place.

> However, the example presented makes it seem as though you really don't
> need Deferreds, because the only problem with the single-callback approach
> is handling errors. Another major motivation is the ability to return a
> Deferred through a system with several layers, changing the return value at
> each layer by post-processing it a bit.  (One possible example: a REST API
> that wants to deal with objects, and goes via a translation of [bytes from
> HTTP]->[JSON dicts/lists from parsing those bytes]->[domain-specific objects
> by converting JSON objects according to the particular API's spec].)

I never really thought about it before. I just realized right now,
writing this email, that things like DeferredList aren't cleanly
possible if the callback is tied to the request.

Additionally, is the showcase of this in the fixup with "Multiple
Callbacks" I did fine? The SQL to HTML example that was there before
seemed a bit contrived, and I wanted to showcase it in a runnable
snippet that required Twisted. We can't have them install a SQL
server, so I used xml.minidom instead of lxml, even though know it's
complete crap. When I'm done, I should replace the www.example.com
URLs with files hosted on the Twisted doc site. Is there a way to
point to generate a URL like that with Lore?

> However, I think the need would be better illustrated with examples that can
> actually be run than with fake examples where we assume that the user knows
> how something like gevent works.  (Also: gevent doesn't actually work this
> way, for fetching web pages at least, so your example is wrong.  See
> <http://www.gevent.org/intro.html#monkey-patching>.)

I've replaced them with "Library A" and "Library B" placeholders. :)

> It's pretty easy to write a fake implementation of 'fetchWebPageAsync' which
> squirrels away the callback somewhere that the example can call it later,
> and explain that with some handwaving where we say "and pretend that was
> actually some networking code fetching it".  For that matter, the reactor is
> introduced too early in the existing docs; we should demonstrate calling the
> callback synchronously, and then only later introduce a callLater.

I ripped out all the reactor code in the starting sections of the
fixup on purpose, and showcased it with what I think is a clear
example. When introducing the techniques afterwards, I'm going to
gently ramp up the reactor code to be a bit more real-world.

> Anyway I hope this wall of text did not discourage you - I just think you
> need some clearer goals for improving specific aspects of the documentation,
> and you should write those down first before trying to actually address them
> with more docs.

This email did exactly that. Thanks so much!

P.S. I still have a *lot* to learn. Some tutoring on the subtleties of
LoopingCall and coiterator/cooperator would be nice. I'm not going to
even bother to try to explain how inlineCallbacks works to my brain
right now. I've said enough erroneous fact in IRC. I'm going to need a
lot more help in the future.

> Thanks for your time,

I appreciate your time a lot more. You have things to do. I don't.

> -glyph
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python