+$400 docstrings (was Re: [Twisted-Python] Motivations of Twisted Developers & $200 Docstring) Challenge

[glyph at divmod.com]

On Mon, 20 Mar 2006 11:30:51 -0800, Ed Suominen <general at eepatents.com> wrote:

>I'm an active user of Twisted, but even after several years, I'm still
>not sure whether the Twisted developers really *want* you as a user or
>not.

We need more developers, especially those who can make some kind of predictable time investment.  More non-developer users in the F/OSS world mainly help you track down defects faster, and right now we know about waaay more defects than we can handle :).

However, realistically the only way to get more developers is to get more users and then recruit from that userbase.  So, the project is in a bit of a conflicted state now.  We want more users but only the users that are likely to attract more developers.  People who require free guarantees of stability and such are pretty are pretty close to the bottom of the list.  (Of course, everybody *wants* free guarantees...)

>I'm not saying that with any hostility; I'm actually a big fan of
>Twisted and those behind it. They certainly have been very helpful to me
>with my specific questions on IRC, and in that way it's one of the most
>responsive and "live" F/OSS projects around. Frankly, I think the
>Twisted developers understand Python and the theoretical underpinnings
>of software engineering better than 90% of the people developing with
>Python today.

Why, thank you :)

>On the other hand, though, much of the style of development that goes on
>almost seems *intended* to turn off potential users, such as abandoning
>subprojects mid-stream in favor of a Next Big Thing that remains a
>moving and largely undocumented target for long periods of time, and
>failing, for reasons incomprehensible to me, to simply write even 2-3
>lines explanations of what their New and Improved code is actually
>*doing* in the docstrings that Python so conveniently provides for that
>purpose.

Agreed.  This is a huge problem.

>Ironically, one of the latest examples of this is a document
>extractor under development that is supposed to fix some alledgedly
>grevious flaws in epydoc (undetected by this user, by the way) that
>actually has very little source material to work with from the Twisted
>code itself!

The "grievous flaws in epydoc", just for the record, are that it would no longer run over the Twisted codebase at all.  Zope Interface broke it and that was never fully fixed, and other constructs have come along that would cause it to explode while trying to generate the documentation for Twisted.  Even when it was "working" it required a multi-page litany of monkeypatches to get through a complete run.  It was quite detectable by a user that the code had no generated API documentation ;).

However, an alpha of epydoc 3 is out, and it no longer has to import modules to generate documentation, so perhaps this situation is only temporary?

>Another most unfortunate example is that the most powerful way to do
>networking with Python is without a recommendable HTTP server. The
>Twisted devs won't recommend twisted.web because it is "hopelessly
>broken" yet they have nothing released to support it. Twisted web2
>remains largely undocumented and the developers advised those using it
>to do so at their own peril. Note to developers: serving HTTP is the
>biggest single use-case for Python networking!

We're aware of this.  Basically the web situation is this: the web1 server is problematic for a variety of reasons (no support for Deferreds, uploads are fully processed before requests are dispatched to user code, no HTTP/1.1 support, which means no pipelining, and no proper proxying).  The web2 server is remaining unstable because A) it's still desperately inefficient, and B) we all know we made a hash of HTTP the first time around, and so we're trying to get at least the *known* problems fixed for web2 so that we don't go around this wheel of pain again.

It does leave users in an unfortunate position for the time being, and believe me, I am definitely one of those users, and I directly feel this pain.  Divmod's product suffers directly from the web2 rewrite.  Right now we're using twisted.web 'classic' because it's more effort than it's worth to make Nevow work with t.web and t.web2 at the same time.  We want to do things that require the new code (like WebDAV, and tracking the progress of as-yet-incomplete uploads) but we are also hurting for performance at just about every layer and don't want to upgrade until things like streams.

So, if I could think of an easy solution to this problem, I already would have :).  James Knight has been the driving force behind web2 so far, and I admire his efforts, but he's just one guy (with the occasional help of David Reid) and he's not even using this code at work.  Again, there is a labor shortage here.  It's not even a funding shortage in this case: everyone who should be working on web2 has other jobs.  The best way to get this done, honestly, would be for 100 people to join the Divmod fan club and all say "WEB2!!!" in the next meeting :).

>I've complained about these issues for a long time. Now I'm going to put
>my money where my mouth is. I hereby commit to a two hundred dollar
>($200) payment to Twisted, c/o the exarkun at divmod.com account listed
>under the "Contributing" heading of http://twistedmatrix.com/trac/, in
>exchange for one or more of the Twisted developers committing SVN/trunk
>changes in the next two weeks that implement at least 100 meaningful
>class or method docstrings in new or active portions of the Twisted API.
>(I'd personally appreciate a focus on twisted.web2 as that's what I'm
>spending my time on right now, but this challenge is open to any active
>parts of the API.)

I still have $400 in contributions left over from my last birthday ( http://glyf.livejournal.com/2005/05/20/ ) set aside for improving Twisted.

>That works out to $2 per docstring, and those "in the know" should be
>able to write them in just minutes apiece. Spend five hours writing all
>of them (improving "your" parts of Twisted in the process) at 20
>docstrings per hour and pocket $40/hr for your efforts. It sounds like a
>pretty good deal to me, and hopefully to the Twisted devs as well.

That seems like a pretty good rate.  I'll add to that deal.  For each additional 100 docstrings, I'll give an additional 200 dollars directly to whoever does it.

>If this challenge is accepted and it works out well, I might even do it
>again -- I'm *that* interested in the ultimate success of this sometimes
>wonderfully crazy library/framework/Integratotron*

Thanks a lot :)