[Twisted-Python] Re: In Defense of Taps

Andrew Bennetts andrew-twisted at puzzling.org
Thu Feb 13 07:45:50 EST 2003

On Thu, Feb 13, 2003 at 04:26:27AM -0600, Glyph Lefkowitz wrote:
> > Don't try make people swallow all of Twisted at once... it's just too big.
> I don't think the core is too big, and making this argument sounds like we're
> really making users swallow ALL of twisted.  We're clearly not.  I really doubt
> that anyone who can't fit the basic concepts of Reactor, Factory, Protocol and
> Transport into their head within a day or two is competent to do network
> programming in any environment: Twisted is one of the easiest.  Users
> unfamiliar with networking concepts may take longer to get adjusted, but such
> is life and learning.

I agree that comprehending the "basic concepts of Reactor, Factory, Protocol
and Transport" is not too much to ask of a Twisted newbie, so long as you
don't unnecessarily distract them :)

Anyway, I've taken a look at the current shape of the docs, and I've got
a couple of ideas:

  - The "The Basics" chapter covers too little too thinly to be of any use.
    Its pages should be moved accordingly:
      - Installation -> Introduction
      - Bringing up a web server -> Web Applications
      - Using the utilities, Using manhole to configure twisted, Using
        telnet to configure twisted -> a new chapter "Administering Twisted
        Applications", or something like that.  The new chapter would be
        placed somewhere after Low-Level Twisted.
  - Removing the "The Basics" chapter would move the High-Level Twisted to
    the second chapter, right after Introduction.  This feels right to me :)

Currently, "The Basics" chapter offers *nothing* relevant to a newbie
Twisted developer.
> Outside of the set of things you've talked about here, Twisted also contains
> servers for HTTP, NNTP, FTP, IRC, [...], and a plethora of application-domain
> abstractions to support those protocols.  Putting more than one of _those_ into
> a general introductory document because you think it's cool is probably
> overkill, although it's hard to avoid completely.

Definitely.  Ideally, there would be a "Twisted Protocols" chapter in the
docs, with a section for each module in twisted.protocols.* -- but most of
those modules are moving targets, and the API docs aren't too bad, so it's
not a big priority.  Perhaps a limited "Brief Guide to Twisted Protocols"
which just covers a couple of the major modules would be beneficial, though?

> However, TAPs are one of the few base mechanisms for services to communicate
> with each other within the Twistedverse.  Much like Deferreds, they don't imply
> any particular functionality of your application beyond "asynchronous
> execution".  They do have the additional overhead of "persistence", but you
> don't need to worry much about that.  If you're not instantiating any
> particularly weird objects of your own, the framework won't burden you with
> unpersistable stuff at least for the initial TAP construction.

They're orthongal to the task of writing a server/client using Twisted, thus
they shouldn't be up there with Factories, Protocols and Deferreds in the
set of Stuff A New Twisted Developer Needs To Know.

I do agree that TAPs are something a medium-level Twisted programmer should
know, I just disagree with making them a focus of the introductory docs.

> > I'll even go so far as speculating that this will cause more external
> > projects to use TAPs, because they won't be seen as some annoying idea in the
> > beginner docs that they had to skip past to get real work done.
> Based on my experience thus far, I doubt this very much.  I'd be interested in
> a simple show of hands that indicates whether this is an accurate perception.

Me too; after all, it's mere speculation on my part :)

> Most people, including software developers, will skip anything they don't
> already believe they need to read.  Server software developers don't think very
> much about integration: though someone using Twisted has eschewed custom and
> decided not to rewrite everything from scratch, there is still a persistent
> belief in the application developers have an unrealistic degree of control over
> the deployment environment, which turns software immediately into
> consultingware and ruins its general applicability.  TAPs are one tiny, tiny
> microcosm of this problem, so I don't expect that they will solve all the
> world's ills in this regard, either :).

Especially considering they don't yet do WinNT services, for instance :)

I also think that TAPs aren't particularly good for, say, a GTK IRC client.
It makes much more sense for a user to run "/usr/bin/twisted-chat", than
"mktap twisted-chat; twistd -f twisted-chat.tap".  So I'm not convinced that
they're as universally applicable as Factories, etc, which is why I think
they should be less prominent.

I also think that this sentence from plugin.html is worded too strongly
(especially given that it's in <strong> tags):
    "Although there are other ways for Twisted to call your code, all
    Twisted projects should start as a plug-in of some kind."

Twisted is quite capable of being useful standalone apps like GUI IRC
clients, but nothing in that doc suggests to me that writing one as a
Twisted plugin would be a good idea.  Perhaps what I'm really trying to say
is that not all the world is a long-running unix daemon with interesting
persistable state, but currently the docs assume that.

> Featuring the TAP plugin howto prominently at least stimulates debate about
> TAPs.  

Couldn't you just post to the mailing list like a normal person? ;)

>        This conversation is not the first such one (though it is by far the
> lengthiest and most public). Conversations stimulate development in turn.

Indeed! :)

> Despite occasional fits of philosophical obscurantism, I am also more a
> programmer than an ideologue.  While this conversation has been going on, I've
> been kicking around more and more ideas for COIL and how to make this whole
> plugin thing a really slick whiz-bang experience.  I'm sure that will be more
> convincing than an email thread.


> Those of you who are disagreeing, consider that even your disagreement is a
> contribution :-).
> > Give Twisted plugins its own, comprehensive doc that can assume knowledge of
> > basic Twisted, rather than short-changing it by making it the introductory
> > doc.  It could start with a section titled "Motivation", explaining the
> > advantages of twistd and mktap.
> While I think of this as an important introductory doc, it's already the 18th
> document in the book.  In fact, it comes after "Using Dirdbm", which I think
> may be shortchanging it a *bit* more than treating it as introductory :-).

I must admit, I recalled it being more prominent than that... being dumped
after Using Dirdbm is a bit harsh :)

> As to comprehensiveness: I'd love to have a whole chapter on plugins, but
> frankly there isn't that much to cover.  Do you feel that the current doc is
> not thorough enough?
> And finally, motivation: the section "Twisted and You: Where Does Your Code Fit
> In?" may not be optimal, but I think it explains in overview terms why one
> would want to use a TAP.

You're probably right... the existing docs for plugins are fairly good.  The
main part lacking is something that points out "Hey, if you do this, you can
use twistd(1), and take advantage of its daemonisation and logging and
etc...".  It's hard to see the immediate benefit of mktap and twistd, when
you could ignore them and just run "./foo ...", rather than "mktap foo ... ;
twistd -f foo".

So, I'm asking for a more pragmatic motivation, rather than the existing
(but still valuable) philosophical one, which doesn't point this out.
"Framework" vs. "library"?  Who cares.  "Yet another abstraction I have to
cope with"?  Bleagh.  "It will give me proper daemonisation and logging and
..."?  Hey, tell me more!


More information about the Twisted-Python mailing list