[Twisted-Python] Re: Teach Me Twisted Redux

Phil Christensen phil at bubblehouse.org
Thu Mar 20 14:27:19 EDT 2008

On Mar 20, 2008, at 11:36 AM, Glenn H Tarbox, PhD wrote:
> One of the biggest (glaring?) issues with Twisted is the abysmal state
> of the documentation (none) making the code the best source... and
> history is replete with the massive successes that approach has  
> borne...
> What documentation there is includes: "If you need to call a method  
> that
> returns a deferred within your callback chain, just return that
> deferred, and the result of the secondary deferred's processing chain
> will become the result that gets passed to the next callback of the
> primary deferreds processing chain"
> Now, the above is true and clear to those of us who know twisted...  
> but
> I've used that quote for levity... its simply incomprehensible... but
> absolutely critical to understanding the power of twisted.  I'd say  
> that
> the current state of twisted documentation is in part represented by
> that quote... and much of the reason twisted gets thrown out early  
> as an
> option.

I think there's a lot of truth to this, but you can say the same thing  
about a vast number of open source projects. Not that it's an excuse,  
of course, but in most of the projects I can think of that have  
sufficient documentation, such a thing didn't occur until the software  
reached a critical mass of users.

The Twisted community is growing all the time, but it's still just a  
part of the Python community, which despite a number of high-profile  
Python projects, is still eclipsed by Java, Perl and (*shudder*) PHP.

As for the complexity of docstrings, I don't think anyone should be  
considering source code documentation equivalent to a formal tutorial,  
so I think the brevity (and assumption of existing knowledge) is  
pretty much warranted in that case.

> Clearly, the issues Twisted addresses are non-trivial requiring an
> appreciation of the problem space before considering Twisted as a
> solution (you gotta know there's a question before someone tells you  
> the
> answer)... unfortunately, the barrier to Twisted entry at that point
> makes most walk away.  I've seen lots of threads concluding, simply,
> that twisted looked interesting but was simply too dense to even get
> started with... so, they go ahead and roll their own solution,
> inevitably identifying the issues which form heart of the twisted
> architecture, but being too far along to refactor.  And away we go.

I completely agree with the first half of this paragraph, but I have a  
different perspective on the second.

I've been lurking on this mailing list for years, and what I see most  
often is people who do not even begin to understand the concept of  
asynchronous networking. *This* is the barrier to entry, and it's  
almost always the culprit when a potential user walks away from Twisted.

IIRC, at the time Twisted Python was created, thread support in Python  
was simply awful. Threads were absolutely not a solution for dealing  
with multiple client connections to a server application, and the only  
alternative was asyncore.

As many of you may remember, asyncore was not great, but it *was*  
simple. Writing an asyncore application required knowledge of maybe  
three different classes, so while you were working on understanding  
asynchronous networking, you didn't have to deal with learning a vast  
new API.

The problem I've seen with various responses from seasoned Twisted  
devs to unexperienced developers is that one can only explain why you  
don't want to use any kind of long-blocking call so many times. After  
that, answers start becoming glib and/or overly terse, but I can't  
really blame anyone for that.

There are a few introductory tutorials on asynchronous programming,  
but we could always use more. I almost wonder if it would be  
worthwhile to have an asyncore tutorial as the first step, so that  
users can be exposed to asynchronous programming through a simpler  
model, run into some problems, and then see how Twisted makes them all  
go away.

> Exacerbating the problem is the state of the twisted code base.  The
> core itself is clean, high performance and great.  But, there's a  
> large
> percentage of the code base in various states of decay.  Some clearly
> marked as no longer supported... but most simply marked "undocumented"
> and much of the rest necessitating querying #twisted, hopefully at a
> time when someone is available to answer questions... twisted-web
> appears to be worse from a documentation perspective and I'm one of
> those who chose to "just walk away" when a web framework was  
> necessary.

I definitely agree with you about the state of Twisted.Web. I think  
the culprit here is not entirely a shortage of documentation. The real  
problem is Web2.

Now, I am the last person to discourage total or near-total rewrites  
of stable codebases. I know I personally have made the choice to  
rewrite projects instead of fixing the things that are broken on a  
number of occasions, but the creation of the Web2 initiative has  
caused endless problems when explaining Twisted's web features to new  

There's lots of good stuff in Web2, but its existence as a half- 
supported, half-unreleased project causes much confusion to new users,  
probably because they always assume that it's a choice between  
Twisted.Web 1.0 and Twisted.Web 2.0, which is obviously not the case.

The web situation is also exacerbated in some ways by Nevow being a  
separate project under the Divmod banner. Much of the greatness of a  
Twisted web stack (IMO) comes from using it with Nevow, and in this  
AJAX-crazed world we live in, Nevow.Athena could easily sell the whole  
web layer to anyone working on a heavily AJAX-oriented project.

Finally, now imagine yourself as a web developer who has never had to  
deal with asynchronous programming, or Twisted, or Twisted.Web; you  
don't know whether you should use web2 or not, and you've never heard  
of Athena.

Obviously, there's good reasons why these various things have happened  
over the years, but I really think documentation isn't enough to fix  
organizational issues. And, given the state of flux (apparent or  
actual) in the Twisted.Web arena, who wants to write extensive  
documentation that will be completely broken in a year?

(We all know why that line of thought is wrong, but it doesn't make it  
any less common...)

> For example, I've been playing with Twisted for a while now and only
> recently stumbled upon AMP... perhaps thats a personal issue and I do
> have fundamental intelligence limitations... but perhaps its
> illustrative.

I have to say, I still don't really know what AMP is, and I've asked a  
bunch of times. Maybe I actually do know, by now, but I've certainly  

> What we need is a core documentation / presentation / communication
> strategy to communicate what twisted is and a vehicle to support
> education.  Handling conferences is a degenerate case which requires
> extension with malt beverages.

I really think that doing this properly requires a core team of people  
at least as large as the core Twisted dev group. They would need to  
have both a knack for documentation and an understanding of somewhat  
low-level networking concepts, *and* an ability to read source code  

Now, I don't mean to point out obstacles and say it's hopeless, but I  
think this relates to the critical mass I was talking about earlier. I  
think Twisted just needs to reach that level where there's enough  
people who want to write documentation that it gets done.

Other attempts to force the issue haven't really met with much  
success; I know there used to be (perhaps still are) a few bounties  
out there on documentation, but not much has happened in that arena  
since then. (Personally, I am anti-bounty, as I feel -- perhaps  
irrationally so -- that it cheapens the hours and hours of work that  
people have done for free.)

I think the best way for us to address this in the short term is to  
really try to focus on group documentation. The current Trac wiki can  
barely be called collaborative; as a start why not transfer the  
Twisted docs into the wiki, and allow Trac users to modify and update  
this information?

Furthermore, considering the length some of the posts I see here, I'd  
like to suggest that if you're going to spend 30 minutes writing an  
email explaining some concept, why not write a mini tutorial about it  

Even if you just copy and paste the email you were going to send into  
the wiki, it's still a start, and others can work on it from there.

You know what they say about documentation being like sex; when it's  
good it's great, but when it's bad it's still better than nothing ;-)...


More information about the Twisted-Python mailing list