[Twisted-Python] Re: Teach Me Twisted Redux

[Steve Holden]

Phil Christensen wrote:
> On Mar 20, 2008, at 11:36 AM, Glenn H Tarbox, PhD wrote:
>> One of the biggest (glaring?) issues with Twisted is the abysmal state
>> of the documentation (none) making the code the best source... and
>> history is replete with the massive successes that approach has borne...
>>
>> What documentation there is includes: "If you need to call a method that
>> returns a deferred within your callback chain, just return that
>> deferred, and the result of the secondary deferred's processing chain
>> will become the result that gets passed to the next callback of the
>> primary deferreds processing chain"
>>
Well, perhaps we can extend the conference session a little. [Drinks 
whisky].

So what you are saying is that a callback can find itself needing to 
return a result that is not yet immediately available. So it will return 
a deferred, whose callbacks and errbacks will be triggered in the 
standard way when the result becomes available ...

[Drinks more whisky]. Did Itamar write this? HELP!

>> Now, the above is true and clear to those of us who know twisted... but
>> I've used that quote for levity... its simply incomprehensible... but
>> absolutely critical to understanding the power of twisted.  I'd say that
>> the current state of twisted documentation is in part represented by
>> that quote... and much of the reason twisted gets thrown out early as an
>> option.
> 
I believe you've made your point, and one of the reasons I chose Twisted 
as the topic for the session was because the document doesn't show, if I 
may briefly channel Zed Shaw, how truly fucking amazing Twisted is.

The good news: this is a remediable deficiency.

The bad news: it has neither the glamor nor the appeal of hacking at the 
codeface.

Now you have a Foundation, consider having it sponsor (through the PSF 
is you aren't registered as a mentoring organization) someone to work on 
the docs in Summer of Code. Even trawling the mailing lists for useful 
material would yield some gems that could be used to improve things.

> I think there's a lot of truth to this, but you can say the same thing 
> about a vast number of open source projects. Not that it's an excuse, of 
> course, but in most of the projects I can think of that have sufficient 
> documentation, such a thing didn't occur until the software reached a 
> critical mass of users.
> 
> The Twisted community is growing all the time, but it's still just a 
> part of the Python community, which despite a number of high-profile 
> Python projects, is still eclipsed by Java, Perl and (*shudder*) PHP.
> 
Some things are simply better not documented.

> As for the complexity of docstrings, I don't think anyone should be 
> considering source code documentation equivalent to a formal tutorial, 
> so I think the brevity (and assumption of existing knowledge) is pretty 
> much warranted in that case.
> 
>> Clearly, the issues Twisted addresses are non-trivial requiring an
>> appreciation of the problem space before considering Twisted as a
>> solution (you gotta know there's a question before someone tells you the
>> answer)... unfortunately, the barrier to Twisted entry at that point
>> makes most walk away.  I've seen lots of threads concluding, simply,
>> that twisted looked interesting but was simply too dense to even get
>> started with... so, they go ahead and roll their own solution,
>> inevitably identifying the issues which form heart of the twisted
>> architecture, but being too far along to refactor.  And away we go.
> 
> I completely agree with the first half of this paragraph, but I have a 
> different perspective on the second.
> 
> I've been lurking on this mailing list for years, and what I see most 
> often is people who do not even begin to understand the concept of 
> asynchronous networking. *This* is the barrier to entry, and it's almost 
> always the culprit when a potential user walks away from Twisted.
> 
That's true, and I tried to get this across using the analogy of a GUI 
interface to explain the event-driven nature of Twisted. This was fine 
for people who had done GUI programming, but Glyph later gave me an 
email analogy which would work far better for those who hadn't.

One of the distinguishing features of *great* documentation is that it 
provides the same information in ways that make it accessible to 
different audiences.

> IIRC, at the time Twisted Python was created, thread support in Python 
> was simply awful. Threads were absolutely not a solution for dealing 
> with multiple client connections to a server application, and the only 
> alternative was asyncore.
> 
And Twisted is still a way better solution even now.

> As many of you may remember, asyncore was not great, but it *was* 
> simple. Writing an asyncore application required knowledge of maybe 
> three different classes, so while you were working on understanding 
> asynchronous networking, you didn't have to deal with learning a vast 
> new API.
> 
> The problem I've seen with various responses from seasoned Twisted devs 
> to unexperienced developers is that one can only explain why you don't 
> want to use any kind of long-blocking call so many times. After that, 
> answers start becoming glib and/or overly terse, but I can't really 
> blame anyone for that.
> 
Except those who failed to extract the kernel of these three thousand 
answers and incorporate it into the documentation. If it's a failure 
it's a common one, and it's distributed throughout the community.

> There are a few introductory tutorials on asynchronous programming, but 
> we could always use more. I almost wonder if it would be worthwhile to 
> have an asyncore tutorial as the first step, so that users can be 
> exposed to asynchronous programming through a simpler model, run into 
> some problems, and then see how Twisted makes them all go away.
> 
>> Exacerbating the problem is the state of the twisted code base.  The
>> core itself is clean, high performance and great.  But, there's a large
>> percentage of the code base in various states of decay.  Some clearly
>> marked as no longer supported... but most simply marked "undocumented"
>> and much of the rest necessitating querying #twisted, hopefully at a
>> time when someone is available to answer questions... twisted-web
>> appears to be worse from a documentation perspective and I'm one of
>> those who chose to "just walk away" when a web framework was necessary.
> 
> I definitely agree with you about the state of Twisted.Web. I think the 
> culprit here is not entirely a shortage of documentation. The real 
> problem is Web2.
> 
> Now, I am the last person to discourage total or near-total rewrites of 
> stable codebases. I know I personally have made the choice to rewrite 
> projects instead of fixing the things that are broken on a number of 
> occasions, but the creation of the Web2 initiative has caused endless 
> problems when explaining Twisted's web features to new users.
> 
> There's lots of good stuff in Web2, but its existence as a 
> half-supported, half-unreleased project causes much confusion to new 
> users, probably because they always assume that it's a choice between 
> Twisted.Web 1.0 and Twisted.Web 2.0, which is obviously not the case.
> 
> The web situation is also exacerbated in some ways by Nevow being a 
> separate project under the Divmod banner. Much of the greatness of a 
> Twisted web stack (IMO) comes from using it with Nevow, and in this 
> AJAX-crazed world we live in, Nevow.Athena could easily sell the whole 
> web layer to anyone working on a heavily AJAX-oriented project.
> 
> Finally, now imagine yourself as a web developer who has never had to 
> deal with asynchronous programming, or Twisted, or Twisted.Web; you 
> don't know whether you should use web2 or not, and you've never heard of 
> Athena.
> 
> Obviously, there's good reasons why these various things have happened 
> over the years, but I really think documentation isn't enough to fix 
> organizational issues. And, given the state of flux (apparent or actual) 
> in the Twisted.Web arena, who wants to write extensive documentation 
> that will be completely broken in a year?
> 
Nobody. If anyone offers, back away, smiling.

> (We all know why that line of thought is wrong, but it doesn't make it 
> any less common...)
> 
>> For example, I've been playing with Twisted for a while now and only
>> recently stumbled upon AMP... perhaps thats a personal issue and I do
>> have fundamental intelligence limitations... but perhaps its
>> illustrative.
> 
> I have to say, I still don't really know what AMP is, and I've asked a 
> bunch of times. Maybe I actually do know, by now, but I've certainly 
> forgotten.
> 
>> What we need is a core documentation / presentation / communication
>> strategy to communicate what twisted is and a vehicle to support
>> education.  Handling conferences is a degenerate case which requires
>> extension with malt beverages.
> 
> I really think that doing this properly requires a core team of people 
> at least as large as the core Twisted dev group. They would need to have 
> both a knack for documentation and an understanding of somewhat 
> low-level networking concepts, *and* an ability to read source code 
> effectively.
> 
> Now, I don't mean to point out obstacles and say it's hopeless, but I 
> think this relates to the critical mass I was talking about earlier. I 
> think Twisted just needs to reach that level where there's enough people 
> who want to write documentation that it gets done.
> 
> Other attempts to force the issue haven't really met with much success; 
> I know there used to be (perhaps still are) a few bounties out there on 
> documentation, but not much has happened in that arena since then. 
> (Personally, I am anti-bounty, as I feel -- perhaps irrationally so -- 
> that it cheapens the hours and hours of work that people have done for 
> free.)
> 
> I think the best way for us to address this in the short term is to 
> really try to focus on group documentation. The current Trac wiki can 
> barely be called collaborative; as a start why not transfer the Twisted 
> docs into the wiki, and allow Trac users to modify and update this 
> information?
> 
> Furthermore, considering the length some of the posts I see here, I'd 
> like to suggest that if you're going to spend 30 minutes writing an 
> email explaining some concept, why not write a mini tutorial about it 
> instead?
> 
[Is that the sun starting to brighten the night sky over the horizon?]

> Even if you just copy and paste the email you were going to send into 
> the wiki, it's still a start, and others can work on it from there.
> 
> You know what they say about documentation being like sex; when it's 
> good it's great, but when it's bad it's still better than nothing ;-)...
> 
And when you haven't had any for years you will even read the Twisted 
documentation ;-)

regards
  Steve