[Twisted-web] Re: Thoughts on documentation, wading into Twisted, Nevow, etc.

[lloyd at paisite.com]

Hi Valentino,

On Fri, July 7, 2006 8:10 am, Valentino Volonghi aka Dialtone wrote:

> I don't think we can be satisfied but unfortunately IMHO the real problem
> with twisted is not twisted itself but the big shift in the way you should
> think that drives away most people. I know many intelligent and very
> capable programmers that do understand twisted (or are able to) but they
> do not intend to learn it because they feel it's hard while threading
> isn't as hard as we picture it.

This doesn't really get at my mental stumbling blocks. I find the
arguments for adopting the Twisted approach quite compelling -- indeed
they've fueled my interest and attempts to understand. I think I
understand the theory -- if somewhat dimly. But, for me, there seem to be
big gaps in the explanatory connections between theory, application, and
code. I can't look at a snippet of Twisted code and tell you why it was
written the way it was; much less write my own code.

It's quite possible that I need more hand-holding than the majority. And
it may not be of interest to the community, nor simply cost-effective, to
reach down to folks at my level of incompetence. But, look at it another
way, if I can be made to understand through better documentation then you
know you're reaching the widest possible audience.

Think of me as a tough unit test for documentation.

Note that I have not looked at the documentation in nearly a year; it may
very well have improved substantially since then.

> Even writing two alternatives using the two different models
> could probably help.

As I recall, one of the Python books does this.

Code that I can enter, run, and experiment with is definitely helpful in
my case. One of the problems that I have with many of the code examples in
the Twisted docs is that I don't understand what they're intended to do
nor why I would want to do it. May very well be just the widget I need,
but lack of context keeps me in the dark.

I'm referring here to many of the examples in:

http://twistedmatrix.com/projects/web/documentation/examples/

> I don't think I can answer this question. When I first learnt twisted I
> used the documentation and found it pretty good.

Others have said this as well, so clearly the docs are good so far as they
go. The question is, can they be improved to reach a wider audience and,
if so, how? And are there enough generous folks around with the time,
will, and competencies to make it happen?

> Maybe it's too concise. Every word in the current documentation must be
> carefully read because it's meaning is incredibly important to actually
> understand the text. Some things are implicit in the meaning given to that
> word and are never explained.

Amen.

I found that when I tried to track down enlightening definitions of many
of the technical terms in the Twisted documentation I got stuck in
tautological loops. This was a clue to me that some of the documentation
is the language of an in-group who have worked together closely enough and
long enough to have a large body of tacit understandings and assumptions
that have not found their way into the docs.

This is not a bad thing, per se -- as long as the in-group is only talking
among itself. But it does point to a strategy for improving the docs to
make them more accessable to more people.

Years ago, general semantist S. I. Hayakawa proposed the "ladder of
abstraction." Many language constructs, he proposed, can be arrayed on a
"ladder of abstraction," starting at the first rung with, say, a pointing
operation (I point to Bessie the cow grazing in the field) and ranging up
through "Bessie," a symbolic tag attached to the perceptual object; "cow;"
"ruminant;" "livestock;" "agricultural asset;" and "enterprise capital,"
etc. If I'm thinking Bessie and use the term "agricultural asset," you may
picture a tractor in your mind and... we've just miscommunicated by a fair
stretch.

Anyway, this is old stuff. The solution is not to avoid abstractions --
they're powerful and useful -- but to be careful in using highly abstract
words... qualify them by moving down the ladder of abstraction as far as
possible, ideally to a pointing operation -- in our case, an object
diagram or a working code snippet.

> This is another 'defect'. Twisted developers are extremely capable in
> python and they tend to use the language as much as possible. Of course
> this requires the same knowledge from the reader.

I'm not sure I would call this a defect. We want developers to write the
most efficient code possible. But if a world-class programmer needs less
competent programmers to understand and/or support or use the code, then
he/she is obligated to provide contextual documentation that communicates
at  a level that the code consumers can understand. And, if the programmer
can't do it, then he/she would be wise to recruit/enlist competent
documentation writers and spend the time necessary to help them understand
what they need to know.

I understand, of course, that in the all-volunteer open source army, this
is a difficult thing to do.

>
> In my opinion this is a good thing overall, the only problem is that
> sometimes docstrings are missing and this makes reading code harder than
> it could be.

Yes, clearer and more abundant docstrings would definitely help me, at
least. As would better definitions of technical terms, with explanatory
examples, illustrations, diagrams and code snippets where this would help;
and more short application examples with thorough contextual explanations
of why the code was written the way it was.

Many thanks, Valentino, for your thoughtful consideration of the issue.

Do you have the time and will to help me develop more Techno Turkey
Adventures?

Best wishes,

Lloyd R. Prentice