[Twisted-Python] Deferred documentation.

[Glyph Lefkowitz]

On Mar 21, 2011, at 9:30 PM, Jasper St. Pierre wrote:

> On IRC, exarkun, glyph, spiv and idnar encouraged me to do a bit of
> work on the Defer documentation.

Yay!  This documentation could definitely use some work.

> I kept get confused between the things like returning a Deferred from a callback and chainDeferred, which I found out wasn't that useful:

Yeah, chainDeferred is not a great method.  Now that Deferreds are non-recursive, I think it's purely worse than inserting an additional Deferred as a result from a callback.

> My eventual goal is to reduce the number of documentation about defer
> down to a near-impossible two documents. I'm hoping to merge some of
> the good stuff of the other thousands of documents.

That would be absolutely great.

> Thoughts so far?

While I applaud your intent, these drafts look quite rough.  The random interjections and asides in <http://magcius.mecheye.net/twisted/DeferHowTo-Fixup.html> seem distracting and confusing to me.  Trying to put myself in the mind of a newcomer, I find myself asking many questions which are irrelevant to what I'm trying to learn:

What's "async"?  Why is it hard?  (The original document mentions asynchronous stuff, but in the context of full english sentences.)
Where are we going shopping?  What does shopping have to do with this?
What's gevent?  Does this have something to do with Twisted?
What's node.js?  This looks like Javascript, what does it have to do with Twisted, which I thought was in Python?

(Now I've gotten distracted and I'm reading about gevent and node.js rather than making my Twisted application work and completing the Deferred tutorial.  Epic fail.  But, if I were to continue...)

What's "this pattern"?  Functions?  Don't lots of programs use functions?
How do they use it?  Why is it relevant?
Why is Twisted's right hand blue?  (Forget about being a beginner: I honestly don't even get this reference.  Googling seems to suggest it has something to do with symptoms of heart disease and doesn't seem funny or relevant at all.)
Why is the first explanation of what a Deferred is referred to as "technical mumbo-jumbo"? Is this really complicated?  If I am not super good at programming already, should I not be reading this?
What's an "operation"?  Does that mean 'function' or 'method' or some other special thing?  It says "most operations in Twisted return a Deferred"; but I've called lots of functions in Twisted which returned other objects before reading this tutorial, or returned None.  Were those actually Deferreds?
Why do I "not know where this Deferred has been"?  Do Deferreds get dirty or broken somehow when I add multiple callbacks?  Should I avoid that?

The Python examples in the current Deferred Reference are mostly runnable.  The ones that aren't, should be.  The documentation should stress that you can run these examples simply, and encourage the reader to download and experiment with them, and modify them to see what happens when they do things in a different order.

Instead, the "fixup" changes the first example to rely on a fake library, which will raise exceptions if I try to run it, but doesn't actually explain that 'magiclib' isn't real.  This isn't a huge problem in and of itself (it is trying to demonstrate the "wrong" way to do things, after all) but it sets up the expectation that the rest of the examples are fake, too, and I shouldn't bother to run them.

I think the original document has plenty of issues, but these changes look like they've been written for people who already mostly understand Deferreds, but are having trouble catching some of the nuances, and need humor to diffuse their frustration and more examples to illustrate different usage patterns, rather than a fundamentally clearer or better explanation than what was offered before.  That makes sense, since based on what you've said on #twisted, that's basically the position you find yourself in :).  This document is supposed to be a tutorial though, explaining how to use Deferreds to users who really have no idea what they are (despite its unfortunate name, "Deferred Reference" - that should probably be changed).

One thing I think is very good about this attempted rework, though, is the explanation of the motivation for having Deferreds at all, before explaining how they work.  In the current documentation, it's very unclear why we have such an object in the first place, or what the alternatives to it are.  However, the example presented makes it seem as though you really don't need Deferreds, because the only problem with the single-callback approach is handling errors.  Another major motivation is the ability to return a Deferred through a system with several layers, changing the return value at each layer by post-processing it a bit.  (One possible example: a REST API that wants to deal with objects, and goes via a translation of [bytes from HTTP]->[JSON dicts/lists from parsing those bytes]->[domain-specific objects by converting JSON objects according to the particular API's spec].)

However, I think the need would be better illustrated with examples that can actually be run than with fake examples where we assume that the user knows how something like gevent works.  (Also: gevent doesn't actually work this way, for fetching web pages at least, so your example is wrong.  See <http://www.gevent.org/intro.html#monkey-patching>.)

It's pretty easy to write a fake implementation of 'fetchWebPageAsync' which squirrels away the callback somewhere that the example can call it later, and explain that with some handwaving where we say "and pretend that was actually some networking code fetching it".  For that matter, the reactor is introduced too early in the existing docs; we should demonstrate calling the callback synchronously, and then only later introduce a callLater.

Anyway I hope this wall of text did not discourage you - I just think you need some clearer goals for improving specific aspects of the documentation, and you should write those down first before trying to actually address them with more docs.

Thanks for your time,

-glyph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20110322/e67158fb/attachment-0001.htm