[Twisted-Python] API docs - and examples

[Glyph Lefkowitz]

On Fri, 19 Jul 2002 21:04:05 +0800, Mukhsein Johari <arashi1 at pd.jaring.my> wrote:
> On Friday 19 July 2002 13:17, Christopher Armstrong wrote:
> > I was talking to MetaCosm, and he said the template for API docs he uses
> > in his professional projects is like so:

> > The main thing here is "Big Picture", which should give the method/class
> > some context. NOTES is mainly for temporary stuff; It's probably not
> > crucial to be in the docstrings (probably it should just be in near-by
> > #XXX comments). So yeah, I urge people who are writing docstrings to put
> > stuff into context; I'll try to do the same thing. Whether or not you
> > use a similar format isn't really important, but it seems sane enough to
> > me. We'll probably be doing a lot of this during the 0.99.0 cycle, but
> > it's never too early to start improving documentation :)

Does epydoc support @rationale or something?  That would be handy. :)

"Big picture" stuff should probably also have its own directory in the doc/
tree too -- most other projects have a doc/design/ (or similar) which describes
things like this.  My tragic allergy to formal process has prevented me from
doing enough of this so far; however, now that the cognitive dust has started
to settle, I think that I could venture in and write some design documentation.

I don't think this is really important in the 1.0 release process or the other
things I'm working on at the moment, so it will have to wait a little while.
Is it important to do soon?  Remember also that queries that show me which
internal design ideas are blatantly obvious and which require some more
elucidation are helpful in prioritizing them.

> Yes, I fully agree here! Without the big picture it is difficult to determine 
> the contexts of all modules and classes - especially for newbie "twistas". 
> Also, we need some contextual examples in the api docs itself. A full blown 
> example should be (and generally is)  found in doc/examples.

doc/howto/listings is a better-documented source of that sort of thing.

> Speaking of examples, could the author of the examples please add comments and 
> some explanations of the examples?

In general a good idea, but just in case anybody who wants to check in docs is
listening, I'll respond to your particular questions...

> Jelly?

    http://twistedmatrix.com/products/spread

Anything more specific you need to know?

> What _is_ twistd?

Wow.  I was pretty surprised to find that there is no quick answer to this
question so I'll write one here :-).

'twistd' is the Twisted Daemon.  It is a simple tool designed in a UNIX and
command-line friendly way; however, it is portable to many environments
(including Win32, and even Jython!).  twistd can be used to load multiple
formats (pickle, marmalade XML, "AOT" python source) of serialized twisted
application objects and run them.  Most persistent processes in Twisted are run
using the 'twistd' script.  The notable exception to this rule is currently
graphical client programs.

The "big picture" here is that in the future, twistd may have graphical or
other platform-specific cousins which are designed to work with a particular
toolkit or host operating system.  For example, a gtk-specific mainpoint, or a
win32 service GUI which generalizes some tasks.  My goal is that eventually
_all_ Twisted-ly correct programs will be able to plug in to multiple
main-points that are appropriate to different situations.  (My short term
incentive for this is I want a GladeReactor which displays in an open window
all the open connections and some brief statistics on them in real-time; this
would be useful for debugging certain kinds of application.)

> Should there be only one twistd running with multiple "protocol handlers" or
> one twistd running per protocol?

This *really* depends on your application and your site.  In general, if you're
using twistd, you want either

  1. only one port (in this case, normally PB or HTTP) open, accessing different services
  2. multiple ports (SMTP, POP3, DNS...) open, integrating with the same service
  3. some combination of 1. and 2.

I imagine that the most common case is 3., with 1. running a close second.
Even in the case of 1., you generally want to communicate with *other* web or
PB servers, over either web.distrib or twisted.sister.

In some cases, I just use twisted for the one protocol/one process/one service
model; but that's just because I know it well, it's installed on all my
hardware, and it takes about an eighth of a second to configure a new webserver
that does what I need for a particular box ;-).

> With some of the examples, I find myself saying: "What? That's it? It can't be 
> that simple! Am I missing something?" Some explanations in the code 
> (comments) would go a long way to solving this.

No really, it is that simple.

> Some of the newer documentation is quite good in the sense that it hints at a
> lot but its a bit confusing. The stuff about web-widgets and dom
> templates...It's starts out really great but ends before it should! :-)

We're working on it, but thanks for the feedback!  There has been a big push
for more documentation recently, as Twisted is starting to see more general
applicability and some of the important, core APIs are nearing finalization.
We're not quite there yet.

> The widgets.html doc for example. [...]

Widgets is getting slowly deprecated in favor of the new domtemplate stuff.
(In many ways, they do the same thing.)  It still remains to be seen whether
widgets still has some usefulness beyond what the domtemplate/domwidgets
approach has brought; I am going to be working with the web stuff very soon and
I hope to verify this, and make a device-independent version of
web.widgets.Form (and renderers for the various DOM stuff) before completely
deprecating web.widgets.

> Also, what's a reactor? Is there some other doc to explain the concept of
> reactors?

Among others,

    http://www.cs.vu.nl/~eliens/online/oo/I/2/reactor.html

I agree that Twisted needs its own doc, but the Twisted "Reactor" interfaces
are based around the (relatively) well-known design pattern for event handling.
You shouldn't have to worry too much about the Reactor if you're writing an
application using Twisted's high-level facilities.

> Would you use deferreds in a web application? When and why?

Yes.

When: If you have a web application that needs to do anything that will take a little
while and/or require some network communication, Deferreds are the preferred
way to do this in Twisted.

Why: twisted.web.widgets and twisted.web.domtemplate (and therefore, I believe,
domwidgets, though I'm not absolutely sure...) will handle a Deferred as a
return value and Do The Right Thing.  So we make it nice and convenient for
you; the added benefit of this is that your web application doesn't have to
worry about managing lock contention or thread pools.

Things like twisted.enterprise.adbapi, meant to do potentially long-running,
blocking operations, will typically return a deferred, making it even easier to
take advantage of the convenience in the twisted.web deferred handling.

> As you can see, there are problems with the docs. I understand the main 
> twistas are too busy (insane is more appropriate!) with twisted itself, to 
> write more docs.

Insane is right.  I can't even keep up with the commits list myself any more.

> Honestly, I would _love_ to write some docs, tutorials and howtos, if only I
> could understand the thing myself!

Keep these great questions coming, they help.  At least we could point other
newbies at the mailing list archive after they've been answered.  It would be
an even bigger help if you could organize some of the answers and submit them
as docs.

Hope this helps,

-- 
 |    <`'>    |  Glyph Lefkowitz: Traveling Sorcerer   |
 |   < _/ >   |  Lead Developer,  the Twisted project  |
 |  < ___/ >  |      http://www.twistedmatrix.com      |
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://twistedmatrix.com/pipermail/twisted-python/attachments/20020719/7a3e6da8/attachment.pgp