[Twisted-web] Re: Thoughts on documentation, wading into Twisted, Nevow, etc.

[Terry Jones]

I'm not sure which of the many messages to followup on, so I've picked
LLoyd's summary. Below are a couple of thoughts on the recent posts.

First of all, I wasn't meaning to single out the core of Twisted when
talking about documentation. Like some others, I also find the core to be
relatively well documented. But there are a variety of additional
components (as well as parts of the core, I assume) that seem much less
well documented, and some parts that are now deprecated or discontinued
but, confusingly, for which apparently current documentation is still
online. So I don't think we should simply focus on the core part of
Twisted.

I thought Cory's comment was good:

Cory> Just a final thought: there's still lots of areas that *aren't*
Cory> adequately documented by anyone's standards.  There are areas not
Cory> covered by examples, there are areas not covered by HOWTO's.  The
Cory> core is covered, but any new application worth writing probably uses
Cory> stuff from the fringes.  So our hypothetical new Twisted user starts
Cory> out writing her new application, gets a handle on the core stuff, and
Cory> then discovers those gaps in the fringes.  That can be frustrating.
Cory> Everyone uses a different part of the fringes, so the fringes need to
Cory> be covered almost as well as the core.

My feeling is that the documentation falls off pretty steeply. E.g., guard
and sessions in Nevow. There's a lot going on in quard.py. I don't think
this is a particularly fringe issue - in fact it seems likely to be quite
central to many people using twisted.web.

I like the idea of a cookbook with recipes written by various people. That
sounds like it might be the most useful thing to me anyway. I like the
O'Reilly Cookbook series. E.g., the Python and Perl cookbooks. They are
immediately useful, even to a complete newbie. A cookbook distributes
responsibility and allows indefinite expansion. It would be good to have a
cookbook coordinator to set standards on appearance, maintain the cookbook
pages online etc. If someone were really keen, they could approach O'Reilly.

I think it's important that we make an effort to ensure that whatever we
produce does not go out of date. If I (for example) decide to write
something up but I don't commit to maintaining it, then when it inevitably
becomes outdated it begins the slide towards increasing harm and decreasing
good.

Along these lines, and given the power and beauty of Twisted :-), I would
think it should be possible to have a documentation framework in which
example code from a cookbook could be verified as still working. Of course
this requires more time and programmer effort. But it's already clear that
better documentation will have to be a joint effort between people who feel
like helping by writing and people who feel like helping by explaining
things.

One danger along these lines (this is something I mailed Matt Goodall about
wrt Nevow) is that you don't want the framework to obscure clarity. I.e.,
you don't want an all-powerful framework that hides many things, cutting
the examples to the barest elegant minimum, because the poor reader then
has to (potentially) try to figure out how to work backwards and construct
working standalone code given just a fragment from an example.

I'm not saying this is easy - it's not - just that it's a desirable goal.
One way I've done something like this is to have a collection of templates,
and you use make (or other) to cat these together (and test) entire
examples while just including relevant fragments in other documentation.
Again, this needs programmer help to set up.

We should differentiate between trying to help people who want to use
Twisted and people who are shopping around. Both are valid targets for
people who want to write, but we may as well keep the two clearly
separated.

In terms of the latter (people trying to understand what Twisted is, why
they might want it), I think something like the finger example (translated
out of the 1980s) is very compelling. It's one of the things that sold me.
That particular example would benefit from having the new code in each step
highlighted. Look at how much the finger example covers. As I wrote to some
friends a couple of months ago:

    | The following has a great example of the kind of thing you can do
    | with Twisted.
    | 
    | http://twistedmatrix.com/projects/core/documentation/howto/tutorial/index.html
    | 
    | You don't need to read it all of course.  Below is a bullet-point
    | summary of what they do on those pages.
    |
    | Amazingly, the final code is just 225 non-blank lines. If you remove
    | the one-line {class,def,import} statements, and comments, you're left
    | with a mere 118 lines of code. In case it's not clear, the end point
    | is a server doing everything mentioned below, all at the same
    | time. So they wind up handling about 6 different protocols on 6
    | different ports, each one independently pluggable.

You ain't going to be doing anything like that with Django, TurboGears, or
RoR anytime soon - unless I'm greatly mistaken.

So, threading vs async aside, I don't think it's hard to differentiate
Twisted from what I think of as just "web" frameworks - which are all
focused on building web sites.

Glyph - thanks for the vote of confidence, but I can't step up to be the
Nevow documentation coordinator. I just have too much going on, and I think
it would be bad to undertake something like that without feeling like I was
going to commit to continuing for some time, to maintaining it etc. I hope
I can do other things, like write a cookbook chapter, or write code to help
set up a documentation framework, or, my personal choice, use Twisted to
make millions and then just send you cash...

Terry