[Twisted-Python] Re: Teach Me Twisted Redux
phil at bubblehouse.org
Thu Mar 20 14:27:19 EDT 2008
On Mar 20, 2008, at 11:36 AM, Glenn H Tarbox, PhD wrote:
> One of the biggest (glaring?) issues with Twisted is the abysmal state
> of the documentation (none) making the code the best source... and
> history is replete with the massive successes that approach has
> What documentation there is includes: "If you need to call a method
> returns a deferred within your callback chain, just return that
> deferred, and the result of the secondary deferred's processing chain
> will become the result that gets passed to the next callback of the
> primary deferreds processing chain"
> Now, the above is true and clear to those of us who know twisted...
> I've used that quote for levity... its simply incomprehensible... but
> absolutely critical to understanding the power of twisted. I'd say
> the current state of twisted documentation is in part represented by
> that quote... and much of the reason twisted gets thrown out early
> as an
I think there's a lot of truth to this, but you can say the same thing
about a vast number of open source projects. Not that it's an excuse,
of course, but in most of the projects I can think of that have
sufficient documentation, such a thing didn't occur until the software
reached a critical mass of users.
The Twisted community is growing all the time, but it's still just a
part of the Python community, which despite a number of high-profile
Python projects, is still eclipsed by Java, Perl and (*shudder*) PHP.
As for the complexity of docstrings, I don't think anyone should be
considering source code documentation equivalent to a formal tutorial,
so I think the brevity (and assumption of existing knowledge) is
pretty much warranted in that case.
> Clearly, the issues Twisted addresses are non-trivial requiring an
> appreciation of the problem space before considering Twisted as a
> solution (you gotta know there's a question before someone tells you
> answer)... unfortunately, the barrier to Twisted entry at that point
> makes most walk away. I've seen lots of threads concluding, simply,
> that twisted looked interesting but was simply too dense to even get
> started with... so, they go ahead and roll their own solution,
> inevitably identifying the issues which form heart of the twisted
> architecture, but being too far along to refactor. And away we go.
I completely agree with the first half of this paragraph, but I have a
different perspective on the second.
I've been lurking on this mailing list for years, and what I see most
often is people who do not even begin to understand the concept of
asynchronous networking. *This* is the barrier to entry, and it's
almost always the culprit when a potential user walks away from Twisted.
IIRC, at the time Twisted Python was created, thread support in Python
was simply awful. Threads were absolutely not a solution for dealing
with multiple client connections to a server application, and the only
alternative was asyncore.
As many of you may remember, asyncore was not great, but it *was*
simple. Writing an asyncore application required knowledge of maybe
three different classes, so while you were working on understanding
asynchronous networking, you didn't have to deal with learning a vast
The problem I've seen with various responses from seasoned Twisted
devs to unexperienced developers is that one can only explain why you
don't want to use any kind of long-blocking call so many times. After
that, answers start becoming glib and/or overly terse, but I can't
really blame anyone for that.
There are a few introductory tutorials on asynchronous programming,
but we could always use more. I almost wonder if it would be
worthwhile to have an asyncore tutorial as the first step, so that
users can be exposed to asynchronous programming through a simpler
model, run into some problems, and then see how Twisted makes them all
> Exacerbating the problem is the state of the twisted code base. The
> core itself is clean, high performance and great. But, there's a
> percentage of the code base in various states of decay. Some clearly
> marked as no longer supported... but most simply marked "undocumented"
> and much of the rest necessitating querying #twisted, hopefully at a
> time when someone is available to answer questions... twisted-web
> appears to be worse from a documentation perspective and I'm one of
> those who chose to "just walk away" when a web framework was
I definitely agree with you about the state of Twisted.Web. I think
the culprit here is not entirely a shortage of documentation. The real
problem is Web2.
Now, I am the last person to discourage total or near-total rewrites
of stable codebases. I know I personally have made the choice to
rewrite projects instead of fixing the things that are broken on a
number of occasions, but the creation of the Web2 initiative has
caused endless problems when explaining Twisted's web features to new
There's lots of good stuff in Web2, but its existence as a half-
supported, half-unreleased project causes much confusion to new users,
probably because they always assume that it's a choice between
Twisted.Web 1.0 and Twisted.Web 2.0, which is obviously not the case.
The web situation is also exacerbated in some ways by Nevow being a
separate project under the Divmod banner. Much of the greatness of a
Twisted web stack (IMO) comes from using it with Nevow, and in this
AJAX-crazed world we live in, Nevow.Athena could easily sell the whole
web layer to anyone working on a heavily AJAX-oriented project.
Finally, now imagine yourself as a web developer who has never had to
deal with asynchronous programming, or Twisted, or Twisted.Web; you
don't know whether you should use web2 or not, and you've never heard
Obviously, there's good reasons why these various things have happened
over the years, but I really think documentation isn't enough to fix
organizational issues. And, given the state of flux (apparent or
actual) in the Twisted.Web arena, who wants to write extensive
documentation that will be completely broken in a year?
(We all know why that line of thought is wrong, but it doesn't make it
any less common...)
> For example, I've been playing with Twisted for a while now and only
> recently stumbled upon AMP... perhaps thats a personal issue and I do
> have fundamental intelligence limitations... but perhaps its
I have to say, I still don't really know what AMP is, and I've asked a
bunch of times. Maybe I actually do know, by now, but I've certainly
> What we need is a core documentation / presentation / communication
> strategy to communicate what twisted is and a vehicle to support
> education. Handling conferences is a degenerate case which requires
> extension with malt beverages.
I really think that doing this properly requires a core team of people
at least as large as the core Twisted dev group. They would need to
have both a knack for documentation and an understanding of somewhat
low-level networking concepts, *and* an ability to read source code
Now, I don't mean to point out obstacles and say it's hopeless, but I
think this relates to the critical mass I was talking about earlier. I
think Twisted just needs to reach that level where there's enough
people who want to write documentation that it gets done.
Other attempts to force the issue haven't really met with much
success; I know there used to be (perhaps still are) a few bounties
out there on documentation, but not much has happened in that arena
since then. (Personally, I am anti-bounty, as I feel -- perhaps
irrationally so -- that it cheapens the hours and hours of work that
people have done for free.)
I think the best way for us to address this in the short term is to
really try to focus on group documentation. The current Trac wiki can
barely be called collaborative; as a start why not transfer the
Twisted docs into the wiki, and allow Trac users to modify and update
Furthermore, considering the length some of the posts I see here, I'd
like to suggest that if you're going to spend 30 minutes writing an
email explaining some concept, why not write a mini tutorial about it
Even if you just copy and paste the email you were going to send into
the wiki, it's still a start, and others can work on it from there.
You know what they say about documentation being like sex; when it's
good it's great, but when it's bad it's still better than nothing ;-)...
More information about the Twisted-Python