<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Mar 25, 2011, at 7:28 AM, Jasper St. Pierre wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Big wall of text incoming. If you're going to read any part of this<br>email, search for *IMPORTANT* and read that part.<br><br>Right now I'm stuck at creating a simple example for "deferred dependencies".<br><br>On Tue, Mar 22, 2011 at 10:03 PM, Glyph Lefkowitz<br><<a href="mailto:glyph@twistedmatrix.com">glyph@twistedmatrix.com</a>> wrote:<br><blockquote type="cite">On Mar 21, 2011, at 9:30 PM, Jasper St. Pierre wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">On IRC, exarkun, glyph, spiv and idnar encouraged me to do a bit of<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">work on the Defer documentation.<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Yay! This documentation could definitely use some work.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">I kept get confused between the things like returning a Deferred from a<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">callback and chainDeferred, which I found out wasn't that useful:<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Yeah, chainDeferred is not a great method. Now that Deferreds are<br></blockquote><blockquote type="cite">non-recursive, I think it's purely worse than inserting an additional<br></blockquote><blockquote type="cite">Deferred as a result from a callback.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">My eventual goal is to reduce the number of documentation about defer<br></blockquote><blockquote type="cite">down to a near-impossible two documents. I'm hoping to merge some of<br></blockquote><blockquote type="cite">the good stuff of the other thousands of documents.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">That would be absolutely great.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">Thoughts so far?<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">While I applaud your intent, these drafts look quite rough. The random<br></blockquote><blockquote type="cite">interjections and asides in<br></blockquote><blockquote type="cite"><<a href="http://magcius.mecheye.net/twisted/DeferHowTo-Fixup.html">http://magcius.mecheye.net/twisted/DeferHowTo-Fixup.html</a>> seem distracting<br></blockquote><blockquote type="cite">and confusing to me. Trying to put myself in the mind of a newcomer, I find<br></blockquote><blockquote type="cite">myself asking many questions which are irrelevant to what I'm trying to<br></blockquote><blockquote type="cite">learn:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">What's "async"? Why is it hard? (The original document mentions<br></blockquote><blockquote type="cite">asynchronous stuff, but in the context of full english sentences.)<br></blockquote><br>Should I mention blocking with something like urllib, then showcase<br>another example library that uses regular callbacks?<br></div></blockquote><div><br></div><div>That sounds like a good idea, actually. Start by looking at it blocking, explain the problem with that; then demonstrate the one-callback-when-you're-done approach and explain some issues with that, and then move on to Deferreds.</div><br><blockquote type="cite"><div><blockquote type="cite">Where are we going shopping? What does shopping have to do with this?<br></blockquote><br>Pop culture reference. Killed.<br></div></blockquote><div><br></div><div>To be clear: I did get that reference, actually, and in another context it would be pretty funny :). Just not in these docs that are supposed to be universally accessible.</div><br><blockquote type="cite"><div><blockquote type="cite">What's gevent? Does this have something to do with Twisted?<br></blockquote><blockquote type="cite">What's node.js? This looks like Javascript, what does it have to do with<br></blockquote><blockquote type="cite">Twisted, which I thought was in Python?<br></blockquote><br>Placeholders.<br></div></blockquote><div><br></div><div>Placeholders for what, though?</div><br><blockquote type="cite"><div><blockquote type="cite">(Now I've gotten distracted and I'm reading about gevent and node.js rather<br></blockquote><blockquote type="cite">than making my Twisted application work and completing the Deferred<br></blockquote><blockquote type="cite">tutorial. Epic fail. But, if I were to continue...)<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">What's "this pattern"? Functions? Don't lots of programs use functions?<br></blockquote><br>I guess this is a bit obvious when you have first-class-functions as a<br>language feature, but it's still a "pattern."<br></div></blockquote><div><br></div><div>My point was that the antecedents get a little ambiguous by that point in the text.</div><br><blockquote type="cite"><div><blockquote type="cite">How do they use it? Why is it relevant?<br></blockquote><br>I guess I'm stupid or slow, but it took a while for me to realize that<br>Deferreds were basically a standardized callback mechanism. It's not<br>really written anywhere on the tin: Deferred was to me a bit of an<br>unobvious name for what it does, and before recently I've always<br>associated it tightly to scheduling and the reactor.<br></div></blockquote><div><br></div><div>No, this is a common problem. I think it would be great to address the definition a bit more comprehensively.</div><br><blockquote type="cite"><div><blockquote type="cite">Why is the first explanation of what a Deferred is referred to as "technical<br></blockquote><blockquote type="cite">mumbo-jumbo"? Is this really complicated? If I am not super good at<br></blockquote><blockquote type="cite">programming already, should I not be reading this?<br></blockquote><br>It was prefixed with "The abstract" before. I put it back to "The<br>abstract". To me, it seems like it's written in a way that makes sense<br>only if you understand what a Deferred is, but it was useful, so I<br>didn't rip it out. I see it as a paragraph that will make more sense<br>as you're reading the article, and once you go back and understand it,<br>there's that happy "snap" feeling as you get the concept.<br></div></blockquote><div><br></div><div>It does seem to be a bit disconnected from the flow of the regular text. Perhaps it would be better if it were laid out as a paragraph, and then each sentence, or clause, were examined more closely, perhaps with an accompanying code snippet to clarify it.</div><br><blockquote type="cite"><div><blockquote type="cite">What's an "operation"? Does that mean 'function' or 'method' or some other<br></blockquote><blockquote type="cite">special thing? It says "most operations in Twisted return a Deferred"; but<br></blockquote><blockquote type="cite">I've called lots of functions in Twisted which returned other objects before<br></blockquote><blockquote type="cite">reading this tutorial, or returned None. Were those actually Deferreds?<br></blockquote><br>Yeah, I need to reword that. How about, "because Deferreds are a core<br>part of Twisted, a lot of functions return them"? No... that's not<br>good either.<br></div></blockquote><div><br></div><div>That's closer, but it should be something more specific than "a lot".</div><blockquote type="cite"><div><font class="Apple-style-span" color="#000000"><br></font><blockquote type="cite">Instead, the "fixup" changes the first example to rely on a fake library,</blockquote><blockquote type="cite">which will raise exceptions if I try to run it, but doesn't actually explain<br></blockquote><blockquote type="cite">that 'magiclib' isn't real. This isn't a huge problem in and of itself (it<br></blockquote><blockquote type="cite">is trying to demonstrate the "wrong" way to do things, after all) but it<br></blockquote><blockquote type="cite">sets up the expectation that the rest of the examples are fake, too, and I<br></blockquote><blockquote type="cite">shouldn't bother to run them.<br></blockquote><br>Right now, it's a placeholder for that magic library that I haven't found yet.<br></div></blockquote><div><br></div><div>I can see why you might want to do that. In the narrative flow between urllib and Deferreds, there is a missing step; you may need to fake that out. I'm just saying that you should be very clear and say "this is a fake example, merely for the purpose of illustration". I think that Tornado has some ugly callback-based stuff, but it would be better to leave this example fake than to try to teach users how to use that.</div><br><blockquote type="cite"><div><blockquote type="cite">I think the original document has plenty of issues, but these changes look<br></blockquote><blockquote type="cite">like they've been written for people who already mostly understand<br></blockquote><blockquote type="cite">Deferreds, but are having trouble catching some of the nuances, and need<br></blockquote><blockquote type="cite">humor to diffuse their frustration and more examples to illustrate different<br></blockquote><blockquote type="cite">usage patterns, rather than a fundamentally clearer or better explanation<br></blockquote><blockquote type="cite">than what was offered before.<br></blockquote><br>*IMPORTANT*<br><br>Are there specific changes you find that could make it harder to read<br>for newcomers?<br></div></blockquote><div><br></div><div>I thought my previous message was a list of those :-(</div><br><blockquote type="cite"><div>The concept of Deferreds isn't hard at all, once you understand what<br>they are. The subtle nuances and bits of glue code that Twisted are<br>the things that can trip someone up, and what I'm still learning. A<br>small amount of very specific use cases for Deferreds happen in<br>real-world code and I'd like to show the support that Twisted has for<br>them built-in.<br><br>My goals for this document are:<br><br> 1) A list of guaranteed rules about Deferreds for reference at any time.<br> 2) An introduction to those rules in a format that doesn't require<br>knowledge of others.<br> 3) Showing techniques or tricks that you can play by "exploiting"<br>parts of those rules in the context of a contrived problem.<br> 4) Showing the built-in support for it.<br></div></blockquote><div><br></div><div>My first goal for this document would be a clear, concise explanation of what a Deferred is and why you need it.</div><br><blockquote type="cite"><div>This should help clear up my writing style a bit. I think in terms of<br>separating abstraction layers; I always try separate a fact or rule<br>from logic or a technique that can follow when you can exploit that<br>fact (feel free to ask the people about 'evolution' in<br>#python-offtopic). I also try to think of the code being very linear<br>when it evolves: a new rule is added, you have a problem, you can<br>exploit that rule with a specific technique, the technique is<br>standardized.<br><br>Example A:<br> PROBLEM: You need to create a Deferred with a known result<br> RULE: Callbacks will continue running after you've called "callback"<br>or "errback"<br> TECHNIQUE: You can create a Deferred, call 'callback' on it and<br>return it, without any tricky business<br> STANDARDIZED: twisted.internet.defer.success<br><br>Example B:<br> PROBLEM: You need to get the results from multiple Deferreds without<br>blocking or too much linearity<br> RULE: More than one Deferred can be created and 'run' at the same time<br> TECHNIQUE: You can add a callback to a Deferred, take the result you<br>get and save it in a list or dictionary<br> STANDARDIZED: DeferredList, gatherResults<br></div></blockquote><div><br></div><div>These seem more like recipes to me than introductory documentation. Maybe they should be really close together, to try to drive the concept home, but it would be good to really get it clear in the reader's mind why they <i>fundamentally</i> need Deferreds, then to cover all the subtle <i>different</i> ways you might need them and how you could use them.</div><br><blockquote type="cite"><div>The hardest part is creating simple, short, runnable code that<br>introduces: a problem that doesn't seem silly, a rule that guides<br>toward the solution, the technique that uses the rule to solve it. It<br>was much easier when I could contrive examples of a network-enabled<br>kitchen: recipes map pretty well to code, especially Twisted async:<br><br> 1) Melt butter in a saucepan. When the butter is finished melting,<br>put cocoa powder in.<br> 2) Meanwhile, beat egg whites and sugar, and cream of tartar.<br> 3) When both are done, put the chocolate mix in a Cuisinart, and<br>fold in the egg whites.<br><br>Here you have dependencies (butter needs to be melted before cocoa<br>powder), multi-tasking (you don't want to wait for the butter while<br>beating the eggs), and a way of knowing when things are done (so you<br>can fold them into the cuisinart).<br><br>( Also, this is a real recipe, ableit simplified and it makes really<br>easy, delicious chocolate mousse:<br><a href="http://articles.latimes.com/2008/feb/13/food/la-fo-watch13recafeb13">http://articles.latimes.com/2008/feb/13/food/la-fo-watch13recafeb13</a> )<br></div></blockquote><div><br></div><div>This looks like a fantastic example. It's comprehensible, concrete, not too long, and involves a strict metaphor for a real world situation, without mixing in any obscure technology. I would be happy if the entire Deferred tutorial were to be structured around it.</div><div><br></div><div>You could also tweak it to introduce additional concepts. For example, errbacks: "If the butter burns...".</div><div><br></div><blockquote type="cite"><div><blockquote type="cite">That makes sense, since based on what you've<br></blockquote><blockquote type="cite">said on #twisted, that's basically the position you find yourself in :).<br></blockquote><blockquote type="cite"> This document is supposed to be a tutorial though, explaining how to use<br></blockquote><blockquote type="cite">Deferreds to users who really have no idea what they are (despite its<br></blockquote><blockquote type="cite">unfortunate name, "Deferred Reference" - that should probably be changed).<br></blockquote><blockquote type="cite">One thing I think is very good about this attempted rework, though, is the<br></blockquote><blockquote type="cite">explanation of the motivation for having Deferreds at all, before explaining<br></blockquote><blockquote type="cite">how they work. In the current documentation, it's very unclear why we have<br></blockquote><blockquote type="cite">such an object in the first place, or what the alternatives to it are.<br></blockquote><br>Once I *knew* what a Deferred was, the other pieces started snapping into place.<br></div></blockquote><div><br></div><div>So it sounds like we're in agreement here: the existing document isn't clear enough about exactly what a Deferred is, it's described too formally and its uses aren't clear enough before we start diving into the technical specifics. Any modification should strive to make it super clear what it is and why you use it.</div><br><blockquote type="cite"><div><blockquote type="cite">However, the example presented makes it seem as though you really don't<br></blockquote><blockquote type="cite">need Deferreds, because the only problem with the single-callback approach<br></blockquote><blockquote type="cite">is handling errors. Another major motivation is the ability to return a<br></blockquote><blockquote type="cite">Deferred through a system with several layers, changing the return value at<br></blockquote><blockquote type="cite">each layer by post-processing it a bit. (One possible example: a REST API<br></blockquote><blockquote type="cite">that wants to deal with objects, and goes via a translation of [bytes from<br></blockquote><blockquote type="cite">HTTP]->[JSON dicts/lists from parsing those bytes]->[domain-specific objects<br></blockquote><blockquote type="cite">by converting JSON objects according to the particular API's spec].)<br></blockquote><br>I never really thought about it before. I just realized right now,<br>writing this email, that things like DeferredList aren't cleanly<br>possible if the callback is tied to the request.<br><br>Additionally, is the showcase of this in the fixup with "Multiple<br>Callbacks" I did fine? The SQL to HTML example that was there before<br>seemed a bit contrived, and I wanted to showcase it in a runnable<br>snippet that required Twisted. We can't have them install a SQL<br>server, so I used xml.minidom instead of lxml, even though know it's<br>complete crap. When I'm done, I should replace the <a href="http://www.example.com">www.example.com</a><br>URLs with files hosted on the Twisted doc site. Is there a way to<br>point to generate a URL like that with Lore?<br></div></blockquote><div><br></div><div>Use 'localhost' URLs and have the user run a 'twistd web' command line for their server; that should be simple enough :).</div><br><blockquote type="cite"><div><blockquote type="cite">However, I think the need would be better illustrated with examples that can<br></blockquote><blockquote type="cite">actually be run than with fake examples where we assume that the user knows<br></blockquote><blockquote type="cite">how something like gevent works. (Also: gevent doesn't actually work this<br></blockquote><blockquote type="cite">way, for fetching web pages at least, so your example is wrong. See<br></blockquote><blockquote type="cite"><<a href="http://www.gevent.org/intro.html#monkey-patching">http://www.gevent.org/intro.html#monkey-patching</a>>.)<br></blockquote><br>I've replaced them with "Library A" and "Library B" placeholders. :)<br></div></blockquote><div><br></div><div>I'm still not really sure you need to talk about other libraries at all, especially not this early in the document.</div><br><blockquote type="cite"><div><blockquote type="cite">It's pretty easy to write a fake implementation of 'fetchWebPageAsync' which<br></blockquote><blockquote type="cite">squirrels away the callback somewhere that the example can call it later,<br></blockquote><blockquote type="cite">and explain that with some handwaving where we say "and pretend that was<br></blockquote><blockquote type="cite">actually some networking code fetching it". For that matter, the reactor is<br></blockquote><blockquote type="cite">introduced too early in the existing docs; we should demonstrate calling the<br></blockquote><blockquote type="cite">callback synchronously, and then only later introduce a callLater.<br></blockquote><br>I ripped out all the reactor code in the starting sections of the<br>fixup on purpose, and showcased it with what I think is a clear<br>example. When introducing the techniques afterwards, I'm going to<br>gently ramp up the reactor code to be a bit more real-world.<br><br><blockquote type="cite">Anyway I hope this wall of text did not discourage you - I just think you<br></blockquote><blockquote type="cite">need some clearer goals for improving specific aspects of the documentation,<br></blockquote><blockquote type="cite">and you should write those down first before trying to actually address them<br></blockquote><blockquote type="cite">with more docs.<br></blockquote><br>This email did exactly that. Thanks so much!<br><br>P.S. I still have a *lot* to learn. Some tutoring on the subtleties of<br>LoopingCall and coiterator/cooperator would be nice. I'm not going to<br>even bother to try to explain how inlineCallbacks works to my brain<br>right now. I've said enough erroneous fact in IRC. I'm going to need a<br>lot more help in the future.<br></div></blockquote><div><br></div><div>Great. I look forward to it :).</div><br><blockquote type="cite"><div><blockquote type="cite">Thanks for your time,<br></blockquote><br>I appreciate your time a lot more. You have things to do. I don't.<br></div></blockquote></div><br><div>I am a pretty busy guy, but Twisted is open source and community driven: you've got just as many bugs to fix in it as I do ;-).</div><div><br></div><div>Please feel free to snip heavily in any replies; anywhere that you feel we've reached an agreement doesn't need more quoting.</div><div><br></div></body></html>