[Twisted-Python] Re: Teach Me Twisted Redux
glyph at divmod.com
glyph at divmod.com
Thu Mar 20 21:29:57 EDT 2008
On 20 Mar, 06:27 pm, phil at bubblehouse.org wrote:
>I've been lurking on this mailing list for years, and what I see most
>often is people who do not even begin to understand the concept of
>asynchronous networking. *This* is the barrier to entry, and it's
>almost always the culprit when a potential user walks away from
Yes. We need more intermediate users to help the beginners. The
problem with the Twisted dev team is that we now have a collection of
amazing, unbelievably gifted, genius Master-level developers, honed by
the continual mutual improvement of the review process. However, people
with such exospheric skills (a layer of the atmosphere named, I believe,
after exarkun's brain) get tired of answering simple, repetitive
questions from impatient and confused novices.
This is a good problem to have, but it's still a problem.
If you know some things about Twisted but you're not really an expert
yet, you should really help others learn by answering questions on this
list and on IRC. It would decrease the pressure on people whose time
would be better spent answering the really, really hard questions, or
better yet fixing bugs so that the hard questions won't even come up.
>IIRC, at the time Twisted Python was created, thread support in Python
>was simply awful. Threads were absolutely not a solution for dealing
>with multiple client connections to a server application, and the only
>alternative was asyncore.
I don't think that Python's thread support was that awful, nor that it
has changed that much since then. Are you referring to the GIL?
Anyway, if you are, let's not talk about this any more.
>As many of you may remember, asyncore was not great, but it *was*
>simple. Writing an asyncore application required knowledge of maybe
>three different classes, so while you were working on understanding
>asynchronous networking, you didn't have to deal with learning a vast
Twisted is actually simpler than asyncore by a fair margin. Compare a
Twisted echo server to an asyncore echoserver. Twisted has _more stuff
in it_ than Asyncore does, but it is a lot easier to use, as well as
more portable, for the application programmer. (You still need to know
about WSA errors on Windows if you want to use asyncore, for example.)
>The problem I've seen with various responses from seasoned Twisted
>devs to unexperienced developers is that one can only explain why you
>don't want to use any kind of long-blocking call so many times. After
>that, answers start becoming glib and/or overly terse, but I can't
>really blame anyone for that.
This is the problem I mentioned above.
>I definitely agree with you about the state of Twisted.Web. I think
>the culprit here is not entirely a shortage of documentation. The real
>problem is Web2.
><snip blah blah blah web web2 confusion etc>
We talked a lot about this at PyCon. I don't want to bury the
resolution here, in the middle of a thread that a lot of people have
probably already killfiled, but this problem _is_ recognized as a
priority and _is_ getting solved, thanks in large part to the renewed
efforts of David Reid, James Knight, and Wilfredo Sanchez.
I'll try to scream it from the rooftops as soon as I can.
>I have to say, I still don't really know what AMP is, and I've asked a
>bunch of times. Maybe I actually do know, by now, but I've certainly
First google hit for "twisted amp":
Perhaps there should be more documentation, but I think that's a good
>I really think that doing this properly requires a core team of people
>at least as large as the core Twisted dev group. They would need to
>have both a knack for documentation and an understanding of somewhat
>low-level networking concepts, *and* an ability to read source code
Absolutely. But - an important note - these people DO NOT need to be
deep Twisted wizards on the level of JP, me, Chris Armstrong or whoever;
they can just have a passing familiarity with the concepts and the
capacity to ask and answer questions when necessary.
>Now, I don't mean to point out obstacles and say it's hopeless, but I
>think this relates to the critical mass I was talking about earlier. I
>think Twisted just needs to reach that level where there's enough
>people who want to write documentation that it gets done.
One of the things I did at PyCon was to reach out to the Django
community and ask them to start considering what it would take for
Twisted to be the preferred container and deployment platform for
Django. They're thinking about it. I hope we can keep that dialogue
Supporting other Python frameworks (especially web frameworks) would
provide us with a much broader audience of people with a different skill
set. Unfortunately 99% of those people will have zero interest in
asynchronous networking, but, for example, "twistd" as a command line
tool is kind of neat. There are lots of ways we can suck them in and
interest them in the wider Twisted universe outside of just web. Bottom
line, though: those are the people who can make our website look great,
who can organize our documentation, etc. It's no secret that I'm not a
fan of Django's technological architecture but I think their
documentation is _fantastic_; easily twice as good as ours, probably
more like ten times as good in certain areas.
>Other attempts to force the issue haven't really met with much
>success; I know there used to be (perhaps still are) a few bounties
>out there on documentation, but not much has happened in that arena
>since then. (Personally, I am anti-bounty, as I feel -- perhaps
>irrationally so -- that it cheapens the hours and hours of work that
>people have done for free.)
Whatever the reason, you're right: bounties never really worked. Divmod
is going to be doing a bunch of consulting for the TSF, but that paid
development isn't going to be on the basis of a ticket-by-ticket basis,
and bounties are not a good way to recognize or compensate contributors.
>I think the best way for us to address this in the short term is to
>really try to focus on group documentation. The current Trac wiki can
>barely be called collaborative; as a start why not transfer the
>Twisted docs into the wiki, and allow Trac users to modify and update
Because the version control system used by trac is broken. Because
trac's wiki syntax is a custom hack which we have no other tools to work
with. Because users who know even the tiniest smattering of HTML can
submit patches against the existing documentation. Because dealing with
spam in trac is a huge hassle. Because trac's wiki isn't included in
the release. Because the tools don't exist *to* include trac's wiki in
the release. Because it's a huge project which would take many man-
months to do nothing but put our documentation into an inferior format.
I could go on.
Maybe you could write a tool which would automatically allow users to
edit the lore documentation and generate patches / file tickets for
reviewing those patches? That would be pretty awesome.
>Furthermore, considering the length some of the posts I see here, I'd
>like to suggest that if you're going to spend 30 minutes writing an
>email explaining some concept, why not write a mini tutorial about it
Writing a tutorial is a lot harder than writing an email.
>Even if you just copy and paste the email you were going to send into
>the wiki, it's still a start, and others can work on it from there.
*this* kind of ad-hoc stuff is what should be on a wiki. But it should
be moving in the direction of the real docs, not away from them.
I hope this was illustrative, if not quite worthy of being a tutorial
More information about the Twisted-Python