[Twisted-Python] Re: Teach Me Twisted Redux

[glyph at divmod.com]

On 20 Mar, 06:27 pm, phil at bubblehouse.org wrote:
>I've been lurking on this mailing list for years, and what I see most 
>often is people who do not even begin to understand the concept of 
>asynchronous networking. *This* is the barrier to entry, and it's 
>almost always the culprit when a potential user walks away from 
>Twisted.

Yes.  We need more intermediate users to help the beginners.  The 
problem with the Twisted dev team is that we now have a collection of 
amazing, unbelievably gifted, genius Master-level developers, honed by 
the continual mutual improvement of the review process.  However, people 
with such exospheric skills (a layer of the atmosphere named, I believe, 
after exarkun's brain) get tired of answering simple, repetitive 
questions from impatient and confused novices.

This is a good problem to have, but it's still a problem.

If you know some things about Twisted but you're not really an expert 
yet, you should really help others learn by answering questions on this 
list and on IRC.   It would decrease the pressure on people whose time 
would be better spent answering the really, really hard questions, or 
better yet fixing bugs so that the hard questions won't even come up.
>IIRC, at the time Twisted Python was created, thread support in Python 
>was simply awful. Threads were absolutely not a solution for dealing 
>with multiple client connections to a server application, and the only 
>alternative was asyncore.

I don't think that Python's thread support was that awful, nor that it 
has changed that much since then.  Are you referring to the GIL? 
Anyway, if you are, let's not talk about this any more.
>As many of you may remember, asyncore was not great, but it *was* 
>simple. Writing an asyncore application required knowledge of maybe 
>three different classes, so while you were working on understanding 
>asynchronous networking, you didn't have to deal with learning a vast 
>new API.

Twisted is actually simpler than asyncore by a fair margin.  Compare a 
Twisted echo server to an asyncore echoserver.  Twisted has _more stuff 
in it_ than Asyncore does, but it is a lot easier to use, as well as 
more portable, for the application programmer.  (You still need to know 
about WSA errors on Windows if you want to use asyncore, for example.)
>The problem I've seen with various responses from seasoned Twisted 
>devs to unexperienced developers is that one can only explain why you 
>don't want to use any kind of long-blocking call so many times. After 
>that, answers start becoming glib and/or overly terse, but I can't 
>really blame anyone for that.

This is the problem I mentioned above.
>I definitely agree with you about the state of Twisted.Web. I think 
>the culprit here is not entirely a shortage of documentation. The real 
>problem is Web2.
><snip blah blah blah web web2 confusion etc>

We talked a lot about this at PyCon.  I don't want to bury the 
resolution here, in the middle of a thread that a lot of people have 
probably already killfiled, but this problem _is_ recognized as a 
priority and _is_ getting solved, thanks in large part to the renewed 
efforts of David Reid, James Knight, and Wilfredo Sanchez.

I'll try to scream it from the rooftops as soon as I can.
>I have to say, I still don't really know what AMP is, and I've asked a 
>bunch of times. Maybe I actually do know, by now, but I've certainly 
>forgotten.

First google hit for "twisted amp":

http://twistedmatrix.com/documents/current/api/twisted.protocols.amp.html

Perhaps there should be more documentation, but I think that's a good 
introductory explanation.
>I really think that doing this properly requires a core team of people 
>at least as large as the core Twisted dev group. They would need to 
>have both a knack for documentation and an understanding of somewhat 
>low-level networking concepts, *and* an ability to read source code 
>effectively.

Absolutely.  But - an important note - these people DO NOT need to be 
deep Twisted wizards on the level of JP, me, Chris Armstrong or whoever; 
they can just have a passing familiarity with the concepts and the 
capacity to ask and answer questions when necessary.
>Now, I don't mean to point out obstacles and say it's hopeless, but I 
>think this relates to the critical mass I was talking about earlier. I 
>think Twisted just needs to reach that level where there's enough 
>people who want to write documentation that it gets done.

One of the things I did at PyCon was to reach out to the Django 
community and ask them to start considering what it would take for 
Twisted to be the preferred container and deployment platform for 
Django.  They're thinking about it.  I hope we can keep that dialogue 
going.

Supporting other Python frameworks (especially web frameworks) would 
provide us with a much broader audience of people with a different skill 
set.  Unfortunately 99% of those people will have zero interest in 
asynchronous networking, but, for example, "twistd" as a command line 
tool is kind of neat.  There are lots of ways we can suck them in and 
interest them in the wider Twisted universe outside of just web.  Bottom 
line, though: those are the people who can make our website look great, 
who can organize our documentation, etc.  It's no secret that I'm not a 
fan of Django's technological architecture but I think their 
documentation is _fantastic_; easily twice as good as ours, probably 
more like ten times as good in certain areas.
>Other attempts to force the issue haven't really met with much 
>success; I know there used to be (perhaps still are) a few bounties 
>out there on documentation, but not much has happened in that arena 
>since then. (Personally, I am anti-bounty, as I feel -- perhaps 
>irrationally so -- that it cheapens the hours and hours of work that 
>people have done for free.)

Whatever the reason, you're right: bounties never really worked.  Divmod 
is going to be doing a bunch of consulting for the TSF, but that paid 
development isn't going to be on the basis of a ticket-by-ticket basis, 
and bounties are not a good way to recognize or compensate contributors.
>I think the best way for us to address this in the short term is to 
>really try to focus on group documentation. The current Trac wiki can 
>barely be called collaborative; as a start why not transfer the 
>Twisted docs into the wiki, and allow Trac users to modify and update 
>this information?

Because the version control system used by trac is broken.  Because 
trac's wiki syntax is a custom hack which we have no other tools to work 
with.  Because users who know even the tiniest smattering of HTML can 
submit patches against the existing documentation.  Because dealing with 
spam in trac is a huge hassle.  Because trac's wiki isn't included in 
the release.  Because the tools don't exist *to* include trac's wiki in 
the release.  Because it's a huge project which would take many man- 
months to do nothing but put our documentation into an inferior format.

I could go on.

Maybe you could write a tool which would automatically allow users to 
edit the lore documentation and generate patches / file tickets for 
reviewing those patches?  That would be pretty awesome.
>Furthermore, considering the length some of the posts I see here, I'd 
>like to suggest that if you're going to spend 30 minutes writing an 
>email explaining some concept, why not write a mini tutorial about it 
>instead?

Writing a tutorial is a lot harder than writing an email.
>Even if you just copy and paste the email you were going to send into 
>the wiki, it's still a start, and others can work on it from there.

*this* kind of ad-hoc stuff is what should be on a wiki.  But it should 
be moving in the direction of the real docs, not away from them.

I hope this was illustrative, if not quite worthy of being a tutorial 
:).