[Twisted-Python] Deferred documentation rewrite

[Glyph Lefkowitz]

On Fri, Jul 31, 2009 at 6:40 PM, Edward Z. Yang <ezyang at mit.edu> wrote:

> You can view an initial draft of the rewrite here:


This is a great first draft!  Very substantial.  I really appreciate you
working on this.

Now I will proceed to rip it to shreds by way of giving you some feedback,
but please try to take this as a constructive review.  I'm happy with what
you've got but given the large amount of dissatisfaction in the community
with the existing Deferred docs, and the widespread confusion that they
cause, I think that these docs have to be totally awesome.


> X Synchronous to Asynchronous: The Method to the Madness
>

I really strongly object to this section title.  Reading through the section
itself, I don't find that it's that objectionable, but one of the
misconceptions that we frequently need to dispel is that Deferreds are
"crazy" or "complex" or "magic".  I think it's very important to reinforce
this for the reader, that this is just an idiom we use for some python
functions to call some other functions in a particular order.

So please get rid of the "madness".

Beyond that, you spend a lot of time talking about *synchronous* and *
asynchronous** *code in this section.  You go so far as to *boldface the
words for emphasis*.  Okay, great, these are important terms, but you're
clearly explaining them as if the user doesn't really know what they mean.
I think that starting with a definition of "synchronous" and "asynchronous"
would be helpful.  Better yet, have an explanation that invokes some code.

The tone also suggests that the user may not quite understand what callbacks
are or how they work.  A brief explanation of higher-order functions in
Python may be in order.  (Or the tone could change to assume that the user *
does* know about this sort of thing, but a little redundancy might not be
amiss here.)

If someone comes to this document with a set of ideas about how network
programming works - for example, that "read()" reads some bytes off of a
socket and blocks until they're ready, it won't be clear how select() and
friends get involved to make this asynchronous programming deal worthwhile.
So it would be useful to explain, at least briefly, how this kind of work
gets done behind the scenes.  You don't want to actually spin up the real
reactor early on in the examples, though.  I think Jonno Lange's document
did a reasonable job explaining how Deferreds interact with the reactor.
It's important to get across that there's no magical interaction, since
that's a considerable source of confusion.  I also wrote an answer on
stackoverflow which addressed this, which might be helpful to you as a
resource:

http://stackoverflow.com/questions/80617/asychronous-programming-in-python-twisted/81456#81456

More minor things:

"tutorial-ish"?  Is this a tutorial or not?  I don't mind some informality
and humor in the documentation, but this is just sloppy.  (Not necessarily
the wording: reading through it, I really can't tell if it's intended to be
a tutorial or not.)

"set of code": this should be "function", or possibly "callback" or
"callable".  It's important to be precise with terminology because later in
the documentation we're going to expect users to know what those terms mean,
and if we've been inconsistent they may be confused.

Throughout Twisted, "Deferred" is used as a noun.  In this document they are
universally referred to as a "Deferred object".  Please drop the "object".

The bulleted lists seem to be a distraction.  Most of them aren't really
enumerating anything, they're just jumping from topic to topic without
finishing a sentence.

You use the word "simple" a lot.  Don't tell me it is or isn't simple:
demonstrate its simplicity.  In one case — "Simple and well defined." —
there isn't even a sentence.

"Asynchronous programming is centered around this notion that:"  This whole
section is very confused.  If it's centered around something, shouldn't it
be one thing?  "this notion"?  Which notion, you've got a list of 3 bullet
points that talk about maybe 5 notions, none of which is an antecedent which
could satisfy "this".

You are throwing lots and lots of examples at the reader, but I find that
users understand better with one thoroughly-explained toy example that they
can pick apart and play with than a whole bunch of abstract stuff.  For
example, "Sometimes I want code to happen during an event, but the event
firing is distinct from my program flow".  A user reading that (if they
understand it) is likely to say "why not start a thread?".  If it is instead
presented in terms of a matter-of-fact "here is what happens" not "here is
why you want this" then the user is more likely to focus on what is
happening (and thus, on understanding Deferreds, which is really the whole
point here) than on whether they *really* actually want it or not.

Hopefully by the time they thoroughly understand it they will know that they
do want it ;-).

- Deferred
>    X Basic operation
>    - Convenience primitives (succeed, fail, execute, maybeDeferred)


This should be covered later on.  "fail" doesn't make any sense unless you
already know about errbacks and chaining.

   ? Callback/Errback chaining


These examples are really weak on explanation.  I won't belabor that point
though, because it seems like you're not really done writing them yet.


>    - Timeouts


Your bare-bones Deferred implementation should really be called something
else.  In Jonathan Lange's example, it was called a "Placeholder".  I can
see readers getting confused about whether Deferreds are something you're
supposed to (or allowed to) implement yourself, or whether they're something
that's a part of Twisted, because you move from talking about your toy
implementation to the real thing without skipping a beat.


> - Composing deferreds
>    - DeferredList/gatherResults
>    - chainDeferred


- Advanced topics
>    - Deferred asynchronous primitives
>    - Sugar syntax
>

I feel like this is throwing too much at the user at once.  It's absolutely
fantastic if you want to address this stuff as well (its docs are even
weaker than Deferred itself), but let's put in a break so that they know
they should go off and try to understand the more basic aspects of Deferreds
before trying to understand gatherResults, inlineCallbacks and
DeferredSemaphore.

More importantly I think you should really focus on getting an extremely
lucid and readable explanation of the core concepts of event-driven
programming and Deferreds before you start adding in these extra bits of
documentation.  Just keep coming back to that.  Pretend you don't understand
why asynchronous programming is useful at all, how select() or non-blocking
I/O works, and read through the document.  Consider whether you understand
what's going on, why these ideas are useful.  Deliberately try to
misunderstand them in a way which is wrong, but consistent with the wording,
and see if you can get to the bottom of your document without being
corrected :).

I have more feedback, but I assume this is more than enough to get you
started :).

P.S. Please CC me in your replies! Thanks.


I'll try to remember, but I'm sure somebody's going to forget - you should
subscribe to the mailing list so you get their messages even if you don't
:).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20090801/079028ab/attachment.htm