[Twisted-Python] Re: In Defense of Taps

Glyph Lefkowitz glyph at twistedmatrix.com
Thu Feb 13 05:26:27 EST 2003

On Thu, 13 Feb 2003 12:14:52 +1100, Andrew Bennetts <andrew-twisted at puzzling.org> wrote:

> Frankly, I see TAPs as one more concept that a new user shouldn't be burdened
> with comprehending.  Reactors, Factories, Protocols, Transports and Deferreds
> are hard enough on their own.  I really think docs that start by using
> reactor.listenTCP, and possibly app.listenTCP, are the way to go.

The metaphor that springs most readily to mind is that of GUI application
toolkits.  Despite the fact that drawing a single pixel is more easily
comprehensible to someone who knows a bit about graphics, and many GTK
programmers are probably graphics people, the tutorial begins with drawing a
whole button.  Buttons may not be of interest to people writing stock-tickers,
and in fact simple drawing primitives and timed callbacks are more useful to
them.  Nevertheless, the "hello world" of GTK is a button you can click on and
a window you can close and stop the main-loop.  The GTK developers want to
encourage you to program in such a way that your code can interoperate with
more of the GTK codebase, and the mechanism that they try to use for that is

As I've said before, Twisted's "widget" analogy is strained, since GTK widgets
are a really *good* abstraction and Twisted mktap plugins are a really *bad*
abstraction, but the general category of thing is the same.

A new Twisted user in any domain except low-level networking should
require only cursory knowledge of Reactors, Factories, Protocols, and
Transports.  Assuming we have 3 classes of user:

    * network protocol developer
    * RPC developer (using PB)
    * web developer

_All_ of these developers should be aware of TAPs and Deferreds, and only the
network protocol developer needs to have a really thorough understanding of
Reactors/Factories/Protocols.  The web developer can get away with only a
rudimentary understanding of Deferreds, too.

I'm talking here about what we need to communicate with these developers, and
not the state the docs are already in: as much as they've improved there's much
more they need.  In particular, we currently imply one path through the
documentation, which I think we've previously agreed is just wrong.  We need
more types of plugins for different interest groups!

> Don't try make people swallow all of Twisted at once... it's just too big.

I don't think the core is too big, and making this argument sounds like we're
really making users swallow ALL of twisted.  We're clearly not.  I really doubt
that anyone who can't fit the basic concepts of Reactor, Factory, Protocol and
Transport into their head within a day or two is competent to do network
programming in any environment: Twisted is one of the easiest.  Users
unfamiliar with networking concepts may take longer to get adjusted, but such
is life and learning.

Outside of the set of things you've talked about here, Twisted also contains
servers for HTTP, NNTP, FTP, IRC, [...], and a plethora of application-domain
abstractions to support those protocols.  Putting more than one of _those_ into
a general introductory document because you think it's cool is probably
overkill, although it's hard to avoid completely.

However, TAPs are one of the few base mechanisms for services to communicate
with each other within the Twistedverse.  Much like Deferreds, they don't imply
any particular functionality of your application beyond "asynchronous
execution".  They do have the additional overhead of "persistence", but you
don't need to worry much about that.  If you're not instantiating any
particularly weird objects of your own, the framework won't burden you with
unpersistable stuff at least for the initial TAP construction.

> I'll even go so far as speculating that this will cause more external
> projects to use TAPs, because they won't be seen as some annoying idea in the
> beginner docs that they had to skip past to get real work done.

Based on my experience thus far, I doubt this very much.  I'd be interested in
a simple show of hands that indicates whether this is an accurate perception.

The feedback I got on the Plugins document at the time I wrote it was that it
was a good grounding in the way that I think about Twisted tying together.  The
people who don't want such an understanding have ignored it to this day :-).
Placing it as an introductory document at least makes this a conscious
decision: they will skip it and ignore it, looking for the document that
explains their particular problem.

Most people, including software developers, will skip anything they don't
already believe they need to read.  Server software developers don't think very
much about integration: though someone using Twisted has eschewed custom and
decided not to rewrite everything from scratch, there is still a persistent
belief in the application developers have an unrealistic degree of control over
the deployment environment, which turns software immediately into
consultingware and ruins its general applicability.  TAPs are one tiny, tiny
microcosm of this problem, so I don't expect that they will solve all the
world's ills in this regard, either :).

Featuring the TAP plugin howto prominently at least stimulates debate about
TAPs.  This conversation is not the first such one (though it is by far the
lengthiest and most public). Conversations stimulate development in turn.

Despite occasional fits of philosophical obscurantism, I am also more a
programmer than an ideologue.  While this conversation has been going on, I've
been kicking around more and more ideas for COIL and how to make this whole
plugin thing a really slick whiz-bang experience.  I'm sure that will be more
convincing than an email thread.

Those of you who are disagreeing, consider that even your disagreement is a
contribution :-).

> Give Twisted plugins its own, comprehensive doc that can assume knowledge of
> basic Twisted, rather than short-changing it by making it the introductory
> doc.  It could start with a section titled "Motivation", explaining the
> advantages of twistd and mktap.

While I think of this as an important introductory doc, it's already the 18th
document in the book.  In fact, it comes after "Using Dirdbm", which I think
may be shortchanging it a *bit* more than treating it as introductory :-).

As to comprehensiveness: I'd love to have a whole chapter on plugins, but
frankly there isn't that much to cover.  Do you feel that the current doc is
not thorough enough?

And finally, motivation: the section "Twisted and You: Where Does Your Code Fit
In?" may not be optimal, but I think it explains in overview terms why one
would want to use a TAP.

 |    <`'>    |  Glyph Lefkowitz: Traveling Sorcerer   |
 |   < _/ >   |  Lead Developer,  the Twisted project  |
 |  < ___/ >  |      http://www.twistedmatrix.com      |
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://twistedmatrix.com/pipermail/twisted-python/attachments/20030213/9dc9d43e/attachment.pgp 

More information about the Twisted-Python mailing list