[Twisted-Python] Twisted Plugins - Implementation Discussion

[Tim Allen]

On Thu, Apr 07, 2011 at 10:35:18AM +1000, Stephen Thorne wrote:
> So the goal of my post to this mailing list is:
> 
> * I would like glyph's goal of having less arbitary code executed at
>   twistd launch time to become a realisation,
> 
> * I would like the process of creating a twisted plugin to be less of a
>   cut+paste+fill-in-blanks hassle.

I notice that Tarek Ziadé's "distutils2" is moving away from "load
a Python module and probe for well-known attributes" toward "define all
metadata in a static file". It sounds like you want a similar thing for
twistd plugins. Perhaps an implementation might look something like
this:

    - At startup, twistd scans twisted/plugin directories on sys.path
      looking for files whose filenames end with '.twistd'.
    - Each such file is loaded with Python's ConfigParser module.
    - Each section in the ConfigParser module represents a plugin whose
      'tapname' is the section name.
    - Each section has a 'description' option, whose value is
      a human-readable string describing the plugin.
    - Each section has a 'module' option, whose value is a string that
      can be passed to Python's __import__ builtin to get a Python
      module.

...where the module defined by 'module' exposes a
'make_service(options)' function, and an 'options' global variable that
is an instance of t.p.usage.Options.

I know you said you didn't like t.p.usage.Options, but I'd be sad to
lose the ability for twistd to support "twistd $PLUGIN --help", and for
that kind of introspection to work, the options data needs to be in
*some* known format. Maybe this might be the time to move to the
stdlib's optparse - or maybe not, now that optparse is (presumably)
deprecated in favour of argparse. Maybe twistd could examine a number of
different well-known variable-names, for different option-parsing
libraries.

Just tossing this out as a strawman for people to point and laugh at.