[Twisted-web] Thoughts on documentation, wading into Twisted, Nevow, etc.

[Jean-Paul Calderone]

On Mon, 3 Jul 2006 03:15:32 +0200, Terry Jones <terry at jon.es> wrote:
>Here's a summary of thoughts I've had over the last month or so. Please try
>to take the following as positive / constructive. I'm happy to be involved
>in trying to alleviate what some people may agree are problems.  Or people
>may not agree that they're problems, which is fine too - more on this
>below.

Thanks for taking the time to write this up, Terry.

>
>A few months ago I started looking at various (mainly Python) frameworks
>with an eye to using one for a project I'm working on. When I found Twisted
>(and friends) I was really happy. I've been writing code for what seems
>like a long time (30 years) and I think I'm pretty critical and demanding
>when it comes to good and clean design. I certainly had not expected to
>find something like Twisted and its various additional components. Without
>having written a line of Twisted code, I made the decision to throw away a
>full year of my own low-level, efficient, debugged, C code (networking,
>custom RPC, argument marshaling, tons of sweat). That was a pretty serious
>step, not taken lightly, and, I still think, a good one.

I'm very happy to hear that you like Twisted, particularly since you
already had such a significant investment in another project at the
time you came upon it.  I'm also glad it means enough to you to take
the time to write an email like this one.

>That's why I'm still here, spending hours writing this email, wanting to
>press ahead, and willing to put in time and effort to help.
>
>In one corner then, we have a great foundation - elegant, dynamic, well
>designed, full of potential, nice programming language, written by talented
>and responsive developers, etc. But... in the other corner, documentation
>that very often seems to be a complete shambles.
>
>There, I've said it.

Actually, I don't think this comes as much of a surprise to anyone. ;)

>
>I don't claim that this is actually a problem. That depends. It may well be
>that I've happened along at a moment which is not optimal for my needs and
>abilities; that there is a well-organized design, development, and
>documentation trajectory; and that I just happened to arrive at a bad point
>on that curve.

I will go so far as to say that it is a problem.  Unfortunately this can't
be blamed on bad timing: if anything, the documentation is in better shape
now than at any other point at which I have been involved with the project.

The project has probably been in better shape with respect to design and 
development planning at various points, but documentation has almost always
been added with little or not forethought to the bigger picture.  The
current plugin document was written because I felt like slacking off work
one afternoon; the producer/consumer document (pitiful as it is) happened
because I was bored one sunday.  I can't really speak for the motivation
behind other writers, but I suspect it is similar in more cases than not.

That said, things have been improving.  The trend is pretty subtle but it
is definitely there.

>I fully sympathize with things being less well documented than they could
>otherwise be.  I love writing code. Following the rise of highlighting
>editors I've even become quite fond of comments. But I'm not even slightly
>fond of writing external docs. I think an important part of getting things
>right in design is to implement something that feels right and then rework
>it until it really feels right and clean, and is also useful. There's
>little point in documenting things before they have settled. Writing code,
>debugging, re-designing, building apps on top of libraries - all those
>things, to me, should take precedence over documentation. I don't believe
>in writing fully fleshed-out specs, docs, and tests ahead of
>implementation.  I even have little time or sympathy for people (like
>myself in this case) who don't have what it takes to use emerging tools
>before they're fully documented for use. I've spent plenty of my own free
>time in the last 20 years writing, giving away, and supporting, code I felt
>like producing.  So I think I understand the equation pretty well.

I can agree with most of what you said above.  The one thing I will disagree
with is not writing tests before the implementation.  Aside from their other
uses, tests can also serve as a form of documentation.  Twisted already
requires tests for all new functionality and all changes to existing
functionality.  Soon it will be required that all tests have docstrings
explaining their purpose.  While this isn't the same as comprehensive prose
describing the motivation and use of a particular API, it can often convey
much of the same information.

>But, Twisted and its related components are released. Valuable (to the
>project) people are trying to use them. This can be frustrating. I think I
>have a reasonably high frustration threshold (but maybe I'm wrong - maybe
>I'm just old & cranky).

True.  Hidden (or not so hidden) here is one of Twisted's biggest problems
right now.  Twisted's developer base is incredibly small, particularly when
considering the scope it encompasses.  Over half of Twisted was developed
by just four people.  With twisted.pb and twisted.web2 one might think that
the load has been spread out a bit more, but other factors have combined to
result in the opposite: over the last year almost 3/5ths of Twisted code
has been written by only three people.  That's a huge amount of effort by
an amazingly small team.

I can't argue that documentation would benefit a great deal of people, but
hopefully it should be clear now why Twisted's documentation isn't better
off than it is.  If anything, I think it's pretty unclear how it is as good
as it is. :)

>It's quite general to talk about "Twisted and its related components".
>There are many moving parts (some conveniently accompanied by moving
>names).  So let me just pick on Nevow - it's perhaps in the middle in terms
>of maturity, documentation, frustration level, etc. I actually don't mean
>to single out Nevow in particular, but it has qualities that make it a good
>example target for discussion. Apologies in advance to anyone this seems to
>pick on - that's not my intention, I have no idea who wrote (or didn't
>write) some of this stuff. I know Matt is involved as he's been kind enough
>to several times send me long explanations and code in email.

Let me clarify Nevow's position a little bit.  Please excuse me if you are
already aware of these details, but I at least want to make sure the archive
contains an accurate description of this.

Glyph started Twisted and accumulated a community of developers.  For a
long time development was performed entirely by volunteers, mostly
scratching a personal itch (ie, no corporate contributions).  There are
a couple exceptions here, but by and large this is an accurate picture.

Later Divmod was started and produced, among other things, Nevow.  Nevow
developed by Divmod in support of a product being developed.  Much of it
went undocumented because the Divmod team knew how it worked well enough
and we had less than no time to spend writing down how to use it for other
people.  Heck, we were already giving it away for free. ;)  Much of what
there is in the way of documentation for Nevow came from the community
after most of the core ideas of Nevow had stabalized.  Unfortunately,
around that point Nevow became complete enough that Divmod didn't need to
invest much time in it (but not complete enough to make all our development
get done overnight by magic elves, so we still didn't have time to document
it) and Nevow started to drift as different parts of the community pulled
it in different directions.  This didn't help the documentation situation
much, since most of the time the only way a new feature was documented was
with another entry in examples.tac.  This went on for a while, as anyone
who has looked at examples.tac might guess.

Lately Nevow has gotten some more attention from Divmod and, while I don't
mean to imply that the directions Nevow was being pulled in by the community
were bad, it was certainly bad that there was no single direction.  A bit
more rudder is being applied now and hopefully in the near future this will
take Nevow to an even greater place than that at which it is now.

These circumstances are all specific to Nevow, of course, and you are talking
about a more general problem.  However, at least you know Nevow's story now,
and anyway it always turns out that whatever project you're talking about has
its own story, too, which explains exactly why it's in the shape it's in. :)

>
>Again, I'm not claiming there's a problem here.
>
>I find Nevow to be extremely promising and yet quite cryptic. Sadly, I've
>built dozens of web sites, many of them commercial, done under contracts
>with real deadlines, budgets, staff, etc. I hate editing ugly templates in
>awkward ugly non-code file formats. There's just so much disgusting stuff
>out there. So to see Nevow and Stan is a real pleasure.  But to try to
>understand it... argh. Hop onto google and try "nevow documentation" and
>read through some other people's experiences (e.g.
>http://www.third-bit.com/pywebblog/archives/cat_twisted.html)
>
>The recent "nevow documentation" thread (kicked off by Manlio, at
>http://twistedmatrix.com/pipermail/twisted-web/2006-May/002652.html) is
>mainly illustrative for its lack of conclusion. I don't find the suggested
>page at http://divmod.org/trac/wiki/TitleIndex to have much (if anything)
>to do with documentation - it seems to be links to code. It's not clear if
>the wiki is more a help or a hindrance (this is probably too condemning,
>but after spending more than enough time looking at things that one later
>discovers are out of date, replaced, deprecated, etc., you start to wonder
>about everything).
>
>Nevow comes with many examples, and that's great. But they're still
>cryptic. E.g., a grep url\\. */*.py in Nevow's examples directory gets me
>over 30 hits, but I don't see that class documented anywhere and when I try
>to use it in my code it doesn't work. E.g., formless is used in many
>examples, but is apparently a dead end. If you disagree with that, you'll
>be reinforcing my sentiment that things are a bit of a mess :-).
>I can give more examples. There's quite a bit that seems very cryptic, and
>I mean beyond the things described in the already very terse and piecemeal
>Nevow documentation.  Of course it takes time to spell things out, but if
>people want them I will. I already decided to take the evening off, have a
>couple of beers, and write this email.
>
>Before trying to finish on a politically correct, upbeat, & constructive
>note, here's a question:
>
>How was it that Twisted (or just Nevow?) managed to lose the attentions of
>Kieran Holland? He (she?) was clearly into Nevow, put a lot of effort into
>understanding things, and yet headed off to Django, which, reading between
>the lines, he/she didn't really think was even nearly up to the
>Twisted/Nevow standard.  That's a bad loss. People who can read and write
>code and who are willing to spend serious amounts of time helping ease
>others into projects are so valuable. The Addendum of Kieran's Stan page
>(http://www.kieranholland.com/code/documentation/nevow-stan/) seems to
>spell things out pretty clearly. It feels like the Nevow (and broader?)
>community has done itself a serious disservice in losing Kieran's attention
>(but, see above comments on whether this is perceived as a problem,
>trajectories, etc).

I don't know the answer to this question.  I am somewhat curious myself.

>To me, totally on the outside, I do not get the impression that there is a
>clear design, code, and release trajectory in place. It feels like there is
>a bunch of very talented designers and coders who are banging out an
>increasing number of cool projects at high speed, leaving a frustratingly
>incomplete, misleading, and atrophying breadcrumb trail of documentation
>behind them.

>From my comments above, you may gather that I don't entirely disagree with
this assessment.  I hope that things are changing, though.

>On to another question: do people think there a problem here, or not?  The
>answer may be no.  I think that's fine, I'm happy to be told to read the
>code and to shut up. At least that would be definitive, and not at all
>unwelcome.
>
>But, if you think there is a problem, what can we do? Here are some
>possible concrete steps that wouldn't take huge amounts of time.
>
> 1) Go through all Twisted & friends web sites and wiki and _ruthlessly
>    delete everything_ that mentions obsolete, deprecated, or renamed
>    projects / modules, broken links, example code using such, etc. There
>    are many of these things, and they are highly misleading and
>    frustrating and give a bad overall impression.

Marking things as deprecated or out of date might be better than deleting
things outright.  On the other hand, any documentation which refers to
code which actually doesn't exist anymore or is known to not work at all
should probably be removed.

> 2) State, where appropriate, that the source code is currently the best
>    documentation. There's no shame in this, and it's much better than
>    having people run across and waste time on obsolete docs, or spend time
>    looking for things that do not exist. E.g., "At this point the
>    nevow.url class is totally undocumented. See examples/{x,y,z} for
>    example usage and the code and comments in nevow/url.py for the best
>    current documentation."

I think this is a good idea.  It seems obvious, but I don't think I've heard
it suggested before.  Of /course/ I always go to the source when I want to
know how something works, and I often recommend that other people do the
same, but having this specifically mentioned /in the documentation/ where
there would otherwise be no mention at all strikes me as very useful.  Just
knowing conclusively that ones search has failed rather than having to keep
looking in case the goal has been overlooked is quite valuable.

> 3) Have someone, preferably an outsider (could be me), write up and
>    maintain a no-punches-pulled overview document about using the various
>    Twisted components. Stick a link to this in a prominent place.
>    http://twistedmatrix.com/trac/wiki/WebDevelopmentWithTwisted is a good
>    start.

This would be quite welcome.

> 4) Have someone go through example code to stick in more comments, get rid
>    of old stuff (no need to rewrite if that takes too long, just throw it
>    away). E.g., the widespread use of formless in Nevow examples. Is it
>    in, is it out?  If the latter, get rid of it before more people die.

This would be useful as well, although I suspect the time would be better
spent writing up real documentation.

> 5) Have someone do some web searches for stuff known to be fixed or out of
>    date, and send mail to the authors asking if they'd please consider
>    updating or otherwise clarifying what they've written, could they add a
>    link or a postscript, to help out future people looking to use Twisted
>    & friends? No need to try to hide the fact that things were a bit of a
>    mess - just aim not to have people get an out-of-date impression or
>    follow links to incorrect / obsolete pages (e.g., links to nevow.com).

This would be useful as well, if it could be done successfully.  I suspect
it is a losing battle, though.  What I would rather see is for more people
to contribute documentation to the project where it could be maintained,
even if the original authors no longer have the motivation to do so.

>I could probably come up with more suggestions. Given the inclinations of
>(I assume) many people still reading this message, no-one is too much
>inclined towards producing actual documentation (and it may not be the
>right moment, as above). So it makes sense to focus on easy hits: purging
>old crud, minimizing wasting the time of people looking to get involved,
>making it clear what the barriers are (e.g., point people at the source,
>tell them that at this point the only way to understand is to ask questions
>(but time is precious) and to read the code).
>
>OK, enough words for now. Thanks again for all the help, all the code, etc.
>
>Terry
>

You're very welcome.  Thank you for the well thought out criticism.

Jean-Paul