[Twisted-Python] Twisted Plugins - Implementation Discussion

Glyph Lefkowitz glyph at twistedmatrix.com
Wed Apr 6 22:54:45 MDT 2011


On Apr 7, 2011, at 12:31 AM, Tim Allen wrote:

> On Thu, Apr 07, 2011 at 10:35:18AM +1000, Stephen Thorne wrote:
>> So the goal of my post to this mailing list is:
>> 
>> * I would like glyph's goal of having less arbitary code executed at
>>  twistd launch time to become a realisation,
>> 
>> * I would like the process of creating a twisted plugin to be less of a
>>  cut+paste+fill-in-blanks hassle.
> 
> I notice that Tarek Ziadé's "distutils2" is moving away from "load
> a Python module and probe for well-known attributes" toward "define all
> metadata in a static file". It sounds like you want a similar thing for
> twistd plugins. Perhaps an implementation might look something like
> this:
> 
>    - At startup, twistd scans twisted/plugin directories on sys.path
>      looking for files whose filenames end with '.twistd'.

While I'm sympathetic to the goal here, I don't like this particular implementation strategy for several reasons.

Right now, in order to properly install a 'twistd' plugin (including those that come with Twisted), you have to do two things:

  1. install some .py files into a package
  2. as the user doing the installation (probably root), run the one-liner at the bottom of <http://twistedmatrix.com/documents/11.0.0/core/howto/plugin.html#auto3>.  If invoking python code in your installation process is too hard, this can be approximated with 'twistd --help 2>&1 > /dev/null'.

This is rocket science.  Nobody can manage it.  Seriously.  After literally _years_ of fighting with conflicting python installation techniques in Debian and Ubuntu, I think that we finally have something that works about half of the time.  I haven't checked up on RedHat in a while and I don't know if they have a working system to do this yet, but they didn't last I checked.

If we invent our own file extension which has to be separately installed, we have to teach distutils, and setuptools, and distribute, and pip, and distutils2, and 'packaging' (as I'm sure that will eventually be incompatible with distutils2 for some silly reason), and easy_install, and dpkg, and rpm, and yum, and apt, and probably five other horrible Python packaging things that I don't even know about yet, how to deal with it.  So I am strongly in favor of keeping everything in .py files and just making a minor tweak to what's stored in dropin.cache (and perhaps allowing dropin.cache to be stored in some location more likely to be writable by individual users, in case the installation process doesn't update it).

Because, frankly, Python installation tools REALLY REALLY SHOULD be able to install Python files into Python packages.  I'm not sure I can make any other assertions quite so strongly.  I'm pretty sure that this is a problem that more than one project is interested in solving.  No other projects are interested in installing '.twistd' files though, I can assure you of that :).

>    - Each such file is loaded with Python's ConfigParser module.

The first rule of the Twisted cabal is of course "don't talk about the Twisted cabal", but the second and possibly even more important rule is "no '.ini' files".  I'd seriously much rather we use XML.  And you can ask Stephen how he feels about XML configuration files.  (Although I'd strongly recommend standing well clear of him when you do that, and making sure that no sharp or otherwise dangerous objects are within easy reach.)

>    - Each section blah blah terrible user interface stuff about 'sections' and other misfeatures of ini files.

I don't want a solution that is hard-coded to deal with the metadata that 'twistd' specifically needs, as Twisted plugins are already used for more than just twistd plugins, and I'd like them to be used for even more.  An interface that limits the metadata to ConfigParser sections would make it awkward to fit into a management GUI or web page.

Plus, the quoting rules for long strings in ini files make it unsuitable for storing long descriptive strings (which is one of the primary use-cases here).

> ...where the module defined by 'module' exposes a
> 'make_service(options)' function, and an 'options' global variable that
> is an instance of t.p.usage.Options.
> 
> I know you said you didn't like t.p.usage.Options, but I'd be sad to
> lose the ability for twistd to support "twistd $PLUGIN --help", and for
> that kind of introspection to work, the options data needs to be in
> *some* known format. Maybe this might be the time to move to the
> stdlib's optparse - or maybe not, now that optparse is (presumably)
> deprecated in favour of argparse. Maybe twistd could examine a number of
> different well-known variable-names, for different option-parsing
> libraries.

For what it's worth, I don't care about this at all.  It's a completely separate issue from the main stuff I care about, and while we should be able to simply delegate to a function that takes a list of strings, I will insist that we fix that separately.

> Just tossing this out as a strawman for people to point and laugh at.

Ha, ha ha ha.  (point).

Thanks for the feedback though.  These are all very common suggestions, and I'm glad for the opportunity to point out why we haven't already done them.

So as not to make this message too long, I'll defer a description my own preferred implementation strategy for a future post to this thread.

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20110407/8cfa7cc7/attachment.html>


More information about the Twisted-Python mailing list