[Twisted-Python] Deferred documentation rewrite
glyph at twistedmatrix.com
Thu Aug 6 00:07:25 EDT 2009
On Mon, Aug 3, 2009 at 6:00 PM, Edward Z. Yang <ezyang at mit.edu> wrote:
> I have updated my draft here:
Thanks. Looks like it's improving. I've got more points to critique now,
but that's only because there's more meat to the tutorial now :).
1. The coding standard in this document is PEP8, not the Twisted coding
standard. Have a look here:
2. "Callbacks are the lingua franca of asynchronous programming" strikes
me as an odd turn of phrase, especially one to open the document with.
3. "This document addresses ... Deferred. It..." - "It" has an ambiguous
antecedent. Are you talking about the document or the Deferred class? Of
course it becomes obvious, but it should be phrased so you don't need to.
4. It's far from obvious what "nonblocking_call" is supposed to be, given
that its definition is "pass". On my first skim through I thought it was a
callback, then had to stop, go back and read again when I realized that
didn't make sense. Brevity is good in examples, but this is too brief.
1. "input" is a builtin function. You might want to avoid using it
for a parameter name.
5. "You might be tempted to define it like this": you're switching
back and forth from second to third person; at first referring to the
reader, then an anonymous different programmer. It might be useful to give
these roles different names; "Alice and Bob" are popular.
6. If you must use a third-person pronoun (as you do the one time you
refer to the API's anonymous user); you should stick to a gender-neutral one
wherever possible, unless of course you're referring to a specific
7. "The Deferred doesn't do anything that you couldn't have done with the
two callback parameters." This isn't strictly true; chaining callbacks, and
dealing with errors that arise in different layers of an asynchronous
callback chain, aren't strictly possible without some additional mechanism.
8. Deferred is mentioned as an API link, Failure isn't.
9. Your explanations of the examples seem backwards. "At its very
simplest, Deferred has a single callback attached to it". I think you
should be explaining the problem being solved by a single callback, since
the synchronous example isn't addressed. The synchronous example obviously
doesn't have a single callback attached to it :). In other words, document
"here's what you might want to do, here's how you can do it" rather than
"here's a thing you can do! by the way, you might want to do it
because...". You've addressed the general why-you-want-to-do-it in the
section above, but it would be helpful to do it in the small for each
10. The DeferredList docs seem wonky in several ways.
1. The opening is hard to follow.
> We are now ready to consider our original problem
what original problem?
> a Deferred that would only fire
"fire"? what does "fire" mean? The term hasn't yet been introduced.
> after some other number of Deferreds fired
Yeah, I'm still not sure what you're referring to. Why would I want
to do this, again?
2. Users really shouldn't be subclassing Deferred themselves, so it's
bad to have an example that does that. Especially one which . The fact
that this is what DeferredList is is an implementation detail,
and an ugly
one at that. Try talking about gatherResults instead, and implementing a
function which does the same thing without a subclass. Or,
perhaps, a class
of your own which just delegates to Deferred for Deferred
than inheriting it.
3. Users *definitely* shouldn't be subclassing Deferred without
upcalling to its __init__. I haven't tested them, but I'm
pretty sure these
examples will just blow up with tracebacks.
4. The examples are never invoked. It's semi-obvious how to use them,
but semi-obvious things are often invoked semi-correctly. Better to have
examples with can be run, or at least ones with a 0-argument entry point
named something like 'start'.
5. "Consider the following interaction of two Deferreds:". You're
setting this up as if it's going to be very formal, but then
is sloppy; you don't name the different deferreds. One of them is "one
deferred", the other is "a Deferred". You don't describe them
independently, the relationship is implicit in the description.
you're describing a fairly complex constellation of objects with
user isn't necessarily familiar yet, you should be clearly labeling the
Deferreds in question in the code sample with variable names
simple as "a" and "b" would probably do fine) and then consistently using
those names to refer to them in the prose as well, so it's easy for the
reader to follow exactly which thing you're talking about. A big problem
with technical documentation, *especially* documentation of Deferreds,
is that it's very easy for a reader to start confusing which thing is
which. Once again, it would be good to set up some kind of
first: *why* are we waiting on multiple Deferreds?
11. "Fluent Interface"? This is more new terminology — terminology
that I am not familiar with, I might add — that isn't defined anywhere in
the document. I think it's more of an appendix than something important to
the main narrative; composing Deferreds, returning a Deferred from another
Deferred, firing a Deferred from another callback, etc, should be covered
1. "Batons" looks like it's going to be more fancy ad-hoc terminology - I
would recommend keeping the language simpler and consistent with other
Twisted documentation :).
12. Still a lot of enumerated lists. Obviously a bad habit to which I am
prone ;-), but when one uses an enumerated list, there should either be in
an expectation that the numbers will be useful. Either, as in this document
review, or code reviews, where the numbers can be used to refer to points in
subsequent discussion, or there's a clear separation of steps. It's not
really clear what the "two possible scenarios" lists are enumerations *of
*. Are they different things that can happen?
13. You should try eliminating the word "consider" from the document.
You seem to have the rhetorical habit, which I've seen from other people
(myself included), of having a sentence which is missing a clear
subject/verb/object relationship, and working around it by saying "consider"
or "let's say". For example, you want to communicate that there's a
Deferred somewhere with some callbacks. You can't just say "A Deferred with
some callbacks.", so you say "Consider: a Deferred with some callbacks", and
now the sentence *seems* complete, but it doesn't really communicate a
Okay, I think that's enough feedback for now. I'll have to do more with
your next round of edits, or my feedback is going to be longer than the
document itself :).
* Why asynchronous?
> - Define synchronous and asynchronous
> - Multiplexing IO
> - Introduce a simple reactor based on select()
> * Why callbacks?
You might want to start with this one, since callbacks are even more
generally useful than asynchronous programming. Your suggestion of a parser
example makes this clear: even if you're parsing synchronously, you'll still
probably have callbacks for different parse rule matches.
> - Asynchronous interaction to synchronous interaction
> - Delocalized execution (the parser example)
> - High level functions in Python review
> Quite frankly, I'm stumped on "defining synchronous and asynchronous."
I'd start with the words themselves. synchronous means "at the same time".
This refers to the timing of the function call and its effects. In a
synchronous program, if I say "read()", then at that same time that "read()"
is called, the reading happens and the data is returned. But, in an
("not at the same time") program, "read()" is called, but its effects happen
This can obviously be fleshed out quite a bit, but I think that core concept
is what's important to communicate.
> it was "you click this button and something should happen!" But that
> is a very different use-case of asynchronous programming than Twisted is.
problem here. One example I like to use to explain why sometimes you just
can't block is this:
button1 = Button()
button2 = Button()
# I need to wait for the user to click on this button
# okay now they've clicked it.
message("Hooray you clicked button 1")
# oh dang, but what if they want to click button 2 first!?!
although you can probably devise a more lucid variant of that :).
One of these days I really want to write a combined Twisted / GTK tutorial
that shows how to ask questions in dialog boxes without blocking and
sub-main-loops and other nasty tricks that GTK programs often get up to in
order to have a question-and-answer UI. Unfortunately, although these
examples do serve as easy-to-identify for learning Twisted programmers, it's
not always immediately clear how this corresponds to networking data, and
the extra complexity of GUI libraries makes it more difficult to run the
And Glyph raised some very salient concerns about what we were trying to
> teach people. I just don't know what direction people are coming from.
I think the best assumption of background for such an introductory tutorial
is to assume that the user doesn't really understand what problem Deferreds
solve, and has thus never done any substantial work in an asynchronous
environment. More experienced users will skim some parts, but that's fine:
more experienced users are easily able to figure out what Deferreds are even
with just the current documentation :).
We shouldn't treat this as a Python tutorial, but it should at least touch
briefly on callable objects and nested variables.
As such, the document now is targeted to "people who know the basics
> of asynchronous programming and grok callbacks", and I've incorporated
> Itamar's excellent suggesting of comparing explicit callback parameters
> and the Deferred object, which I hope dispells the notion of Deferred
> being magical fairly well (my assertion is Deferred is merely an
> abstraction over said callback parameters.) I've also fully fleshed
> out the Deferreds reference; any omissions are my fault.
Again, I think this might be assuming a bit too much. At the very least,
you should find a very, very good tutorial on callbacks and higher-order
functions in Python to point people to as a dependency, so that users who *
don't* have that experience can go read about it somewhere else. (Actually,
every dependency of every document *really* ought to have hyperlinks to
other resources that teach that dependency, so that a user who doesn't know
python but needs to dive into a Twisted codebase will be put on their way
Even people who have some Python experience, but use callbacks rarely, will
often discover there are things they don't know when they start programming
with Twisted and nesting 5 or 6 callbacks in a function. For example, many
people don't know all the fiddly rules of scope nesting. Take a poll of
some potential targets for this intro documentation and ask them if they can
explain why this produces an error:
if x > 3:
x = 2
... but adding 'x=x' to the parameter list of 't' makes it work (although
not like they would expect if they manipulated 'x' in f).
The plan next is to discuss composing deferreds (which will also
> touch on when you should and how to create your own deferreds, as
> well as deferredlist) and the convenience primitives.
I think you need to start talking about creating your own Deferreds, at
least implicitly, very early on in the document. For example, rather than
having "nonblocking_call" be a dummy function, have it maintain a list of
yet-to-complete calls, like this:
pending = 
return "Processed: <" + data + ">"
def nonblockingCall(data, whenSucceeded, whenFailed):
pending.append((data, whenSucceeded, whenFailed))
data, whenSucceeded, whenFailed = pending.pop(0)
whenFailed(RuntimeError("It failed, for some reason."))
then (A) you can demonstrate how the callbacks actually get called in a tiny
little system that the reader can play around with and get comfortable in
before understanding Deferred, and (B) you can illustrate the same example
again with some Deferred logic involved.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Twisted-Python