<br><br><div class="gmail_quote">On Thu, Jan 20, 2011 at 1:55 AM, Glyph Lefkowitz <span dir="ltr"><<a href="mailto:glyph@twistedmatrix.com">glyph@twistedmatrix.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div style="word-wrap: break-word;"><div>Let me begin by saying that I'm _very_ excited about this prospect and I'm looking forward to the output of this project. So, some of what I'm about to say may sound like harsh criticism but <b>please do not let it discourage you</b>. I agree with pretty much everything you're saying about problems with the current documentation. They need to be fixed. I'm offering criticism at this early stage in the hopes that it will be useful and feed into your project, not in the hopes that it will stop you.</div>
<div><br></div><div><div class="im"><div>On Jan 20, 2011, at 1:10 AM, Tom Davis wrote:</div><br><blockquote type="cite"><div>I've been using (and threatening to work on) Twisted for a few years now. It seems like every time I get back into it, I need to dig up old code or Google queries just to get started. Yesterday, Jean-Paul introduced me to the <a href="http://twistedmatrix.com/trac/browser/branches/trial-tutorial-2443-2" target="_blank">trial-tutorial branch</a> which has shed some light on the basics of using trial and testing Twisted client/server applications in general. Until he mentioned it in IRC, I was stuck looking at the actual tests for protocols and deciding which parts of that were generically useful to me. I agreed to finish up that documentation so that it could finally (four years later) be added to trunk (and more importantly, <a href="http://twistedmatrix.com/" target="_blank">twistedmatrix.com</a>). But after thinking about it, I believe the problem runs much deeper than just the lack of a branch merge.</div>
</blockquote><div><br></div></div><div>Yet, the problem _starts_ with a branch merge.</div><div><br></div><div><div>There are many outstanding documentation branches which are substantial improvements, which need to be edited and merged - the trial tutorial among them. It would be great if your efforts could start with getting those landed, turning the crank on the process to get our users better documentation in the interim, as you survey the existing documentation.</div>
</div><div class="im"><blockquote type="cite"><div><div><ol><li>Factories create protocols somehow</li><li>Protocols have methods like connectionMade, connectionLost, and dataReceived to handle events</li>
<li>There are other protocols with other methods. One that I know of, anyway.</li><li>I may need to write a state machine (???)</li><li>I should use an Endpoint or maybe a Service or reactor (but probably not!)</li><li>I should also use Application for serious business</li>
</ol></div><div>Moving forward, howto/clients.html duplicates a lot of these things and fills in some gaps in knowledge while creating more holes. Meanwhile, I still never wanted to create a QOTD or Echo server.</div>
</div><div><br></div><div>I think the point has been made.</div></blockquote><div><br></div></div><div>This kind of tear-down is pretty easy: I'd really like to hear your idea of its inverse, though. What <i>should</i> a newcomer who reads this document know by the end of it? A massive pile of improved documentation would of course be useful, but a good start would be a clear statement of requirements and audience. (As well as an enumeration of <i>different</i> audiences that different documents might serve.)</div>
<div class="im"><br><blockquote type="cite"><div>My <i>real</i> point, though, is that I love Twisted. And I'm constantly wishing it was more accessible to newcomers. Twisted is Python's oldest and most mature event-based networking engine</div>
</blockquote><div><br></div></div><div>(minor nitpick: I really like "event-based" or "event-driven", as you've said here: why does <<a href="http://docs.recursivedream.com/twisted/" target="_blank">http://docs.recursivedream.com/twisted/</a>> say "asynchronous"? I find that especially in documentation it's a lot easier to explain "event-driven", because you can enumerate what the events are, instead of explaining the etymology of "synchronicity"...)</div>
</div></div></blockquote><div><br>I also find that if you use "event-driven" you can compare to event-driven GUI apps, when giving an explanation. Many, many people who have no idea about what "asynchronous networking" is will get the idea pretty easily when you make this comparison. Not everybody, certainly, but a lot.<br>
</div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;"><div><div class="im"><blockquote type="cite"><div>
I want to fix that, among other things. And as luck may have it, I like writing documentation. And I know at least enough Twisted to get the high-level stuff in order and improve the documentation to the point that people will keep reading long enough to make sense of the "idea of Twisted" and be able to implement some basic things and expand upon them later.</div>
<div><br></div><div>There are some things I <b>do</b> want to accomplish early on:</div><div><ul><li>Make the docs accessible (a lot is hidden and hard to find)</li><li>Make them more concise and useful to somebody who doesn't want to know the 50 different ways to skin every cat (including the ones you should never use)</li>
<li>Make them introduce and explain Twisted in a way that somebody as dumb as me can understand it. This means talking about protocols, factories, deferreds, etc. in a way that doesn't require thousands of words of circular explanations, digressions, and duplications.</li>
<li>Document the different Twisted projects as what they are: really useful libraries and/or clients/servers built on top of Twisted that just so happens to also ship with the core. One of them is likely to do what the prospective user wants, too.</li>
</ul></div></blockquote></div><div>These all sound like excellent goals.</div><div class="im"><br><blockquote type="cite"><div><div>There are also some things I <b>don't</b> want to accomplish, at least initially:</div>
</div><div><ul><li>Explain asynchronous programming in depth</li><li>Get into the level of detail that the Krondo series does (but I plan to borrow from it!)</li>
</ul><div>There are things I'll <b>need help with</b>:</div><div><ul><li>The final word on what is/isn't to be used in what context; practical, real-world explanations of why I would use A instead of B (not just "it's more flexible").</li>
</ul></div></div></blockquote></div><div>You will probably have to press us core developers on this one, and you may spark some debates. These tend to sputter out with no clear resolution as everyone is frustrated that nobody's solution (not even their own) is ideal, but you would be doing us all a great service if you really forced us to develop a consensus about certain things (like "what's the best way to build a twisted command-line program", for example) and agree to agree on the current documented "best practice" for those things.</div>
<div class="im"><blockquote type="cite"><div><div><ul>
<li>Better documentation from project creators/maintainers/lovers on their project. These projects make a lot of hard stuff trivial and many do not convey that very well. I'll do my best to reorganize and improve the existing documentation, but most of the projects cater to a specific niche that requires its own core knowledge.</li>
</ul></div></div></blockquote></div><div>The biggest problem with this is that you will find that a very small group of people have created the vast majority of this stuff and don't have time to maintain it all any more :). We certainly don't have a separate dedicated maintainer for each project (although I really wish we could get to that point).</div>
</div></div></blockquote><div><br>I think this right here is the main reason that the docs need to be improved. Easier newbie experience translates to more users translates to more developers translates to more maintainers. Especially if the perception of Twisted as a "languishing" or even "mostly dead" project can be undone. (Yes, I've heard this a number of times. No, I don't know where they get it. I correct it whenever I can. It's something that needs to be addressed, but that's a different conversation.). Fixing up the docs will at least help with this.<br>
</div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;"><div><div class="im"><blockquote type="cite"><div>
<div>Most of this was discussed with Jean-Paul and Glyph at tonight's Python meet-up in Cambridge. Some work has already begun on the new docs here:</div>
</div><div><br></div><div><a href="https://github.com/tdavis/twisted-docs" target="_blank">https://github.com/tdavis/twisted-docs</a></div></blockquote><div><br></div></div><div><div>There have been several abortive efforts to do something grand to re-invent all of the Twisted documentation in the universe, or a complete overhaul of the website, including several false starts that I've made, and most recently the (somewhat arduous, arguably "mostly complete" (fingers crossed on that one)) attempt to do a sphinx migration.</div>
</div></div></div></blockquote><div><br>Seriously man, we're close. 3 more "chunk tickets" in the "edit the lore source" phase (and one of those is finished I think and just needs to be merged). Then another round of "chunk tickets" to manually fix any other little typographical issues in the Sphinx source, and done. I suppose there will also need to be a website deployment process. And probably lots of other minor things that we'll discover as we go.<br>
<br>I've had some major speedbumps, had to find a new job, kids were sick several times, etc. You know...life. I'm getting fired up again though, and thijs seems to be as well, and this coming up now just pumps the bellows.<br>
<br></div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;"><div><div><div>By the way, it seemed tonight that you weren't that familiar with the existing Sphinx efforts, so you may want to have a look at this and see if it is an improvement on our existing documentation index:</div>
<div><br></div><div><span style="white-space: pre-wrap;"> </span><a href="http://twistedsphinx.funsize.net/contents.html" target="_blank">http://twistedsphinx.funsize.net/contents.html</a></div><div><br></div><div>(I was going to point you at the output of the buildbot, but it looks like something in trunk has broken the conversion process. Anyone know what this is about? <<span style="white-space: pre-wrap;"><a href="http://buildbot.twistedmatrix.com/builds/sphinx-html-15724/contents.html" target="_blank">http://buildbot.twistedmatrix.com/builds/sphinx-html-15724/contents.html</a></span>>)</div>
<div><br></div><div>I'm really glad to see that Kevin beat me to the punch on posting his reply though, hopefully we'll see this moving along soon :).</div></div></div></div></blockquote><div><br>I was glad to see just about anyone else interested! :)<br>
<br>It looks like Tom and I have some similar ideas on where we should be going, and I think his Sphinx skeleton is a great example of what things should eventually look like. As I said earlier, I've had a number of similar ideas, though it looks like maybe Tom's are a little more concrete and/or fleshed-out. I just really wanted to get the Sphinx convo "out the door", before I took on another huge project. I'll elaborate in a separate email.<br>
</div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;"><div><div><div>Before you continue too much further in that github repository though, please, consider this:</div>
<div><br></div><div>~/Projects/Twisted/trunk/doc$ find conch core mail names pair web words -name '*.xhtml' -or -name '*.py' | xargs wc -l</div><div> .</div><div> .</div><div> .</div><div><div> 28836 total</div>
<div><br></div><div>That's 28 thousand lines of documentation and sample code. It might not all be the best documentation, but some of it is quite good. Jean-Paul's "twisted web in 60 seconds" tutorial is in there, and that's definitely worth reading. Would you volunteer to rewrite 28 thousand lines of code all at once? Or submit a 28 thousand line patch for review? As problematic as the current situation is, there is a definite potential for some baby vs. bathwater confusion in improving it.</div>
</div><div><br></div><div>I am open to the idea that documentation needs different review practices from code, and there are things about documentation (like narrative flow) which are impossible to automatically verify and may require significant changes at multiple layers simultaneously. These features may not have an obvious parallel in code - which can typically be evolved incrementally, despite the occasional frustrated lament that everything needs to be rewritten.</div>
<div><br></div><div>But, parts of the documentation are code. Another major problem with the current documentation is that it is out of date. We know when code goes out of date because it starts raising exceptions and emitting warnings from our test suite, but we only find out about documentation examples are going out of date when we do a release and some poor soul encounters an error when trying to follow along. I'd really like any effort that undertakes to substantially change the structure of the documentation to make a concerted effort to hit the ground running with test-driven examples which can be automatically verified, so that we have some idea of the impact of code changes on the docs.</div>
</div></div></div></blockquote><div><br>There's a ticket for writing tests for the code samples in Trac.<br> </div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div style="word-wrap: break-word;"><div><div><div></div><div>Also, while you're energized now, please consider what happens to our documentation efforts if you run out of time or out of motivation halfway through this process. Incremental improvements may not necessarily provide the best outcome, but they _do_ mean that users get to see some results from these volunteer efforts even if the original volunteer gets busy with other aspects of life and can't complete their project. Many of your bullet-points above are well suited to individual efforts (some of which are actually issues with the web site, rather than the documentation per se).</div>
</div></div></div></blockquote><div> </div><div>@Tom: agree here. Let's get _some_ benefit out of your energy, rather than you getting discouraged and having half a great set of docs floating out there someplace confusing users. I've been through this myself a couple of times (I've been pretty much ready to "pull the trigger" at least three times!) and have gotten discouraged and/or off track. After a while, my "voom" comes back, but in the meantime, the trunk has progressed, and there's that much more work to do. My advice is to try and get many small changes made, and get them _done_ rather than a few huge changes.<br>
</div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;"><div><div><div></div><div>I'd like to suggest that, at the very least, you create a branch of Twisted (perhaps on Launchpad?) and integrate your work into the doc/ hierarchy of the current tree. That way, at least, it would be easier for someone else to pick up where you've left off and fold things into the right place. This would also provide a way to more easily integrate the doc test suite into the main test suite on our existing build and release infrastructure.</div>
<div><br></div></div><div class="im"><blockquote type="cite"><div>I hope the above came off as optimistic and constructive as I meant it: it's not my intention to insult the documentation or those who wrote it. I just think with a dose of hindsight and a lot of effort, the docs can be awesome. And they would improve the entire Twisted ecosystem along with them.</div>
</blockquote></div></div></div></blockquote><div> Indeed, as the author of many of these original documents, I did not feel insulted in either my person or my work :). Hopefully you didn't feel that way either after reading this reply!</div>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;"><br></div><br></blockquote></div><br>I think this has been pretty constructive on both sides. I look forward to more.<br>
<br>Kevin Horn<br>