[Twisted-Python] Twisted Plugins - Implementation Discussion

[Glyph Lefkowitz]

On Apr 7, 2011, at 12:06 AM, Phil Christensen wrote:

> On Apr 6, 2011, at 8:35 PM, Stephen Thorne wrote:
>> For your consideration, and (constructive) critcism, here is a twisted
>> plugin that is nearly identical to 6 that I have running in  
>> production:
> [snip]
>> serviceMaker = ExampleServiceMaker()
>> 
>> From these 19 lines of code there are 4 things that are relevent:
>> tapname = example
>> description = 'Example Twistd Plugin'
>> options = {'debug':True} if args == ['--debug'] else {'debug':False}
>> service = examplepackage.examplemodule.make_service(**options)
>> 
>> Only 2 of which are relevent for running a twistd daemon other than
>> 'example'
>> 
>> So the goal of my post to this mailing list is:
>> 
>> * I would like glyph's goal of having less arbitary code executed at
>> twistd launch time to become a realisation,
> 
> Makes sense. My first inclination is to suggest creating metadata  
> files that are found via pkg_resources.

We already use a similar mechanism, twisted.python.modules, which uses the same underlying standards as pkg_resources (PEP 302) but is somewhat more flexible.  And we create a metadata file (dropin.cache) which is stored and retrieved using this mechanism.

>> * The entire task of having this python plugin is to link up the
>> metadata with a more or less standard
> 
> Not sure if I get this part. I tend to think of plugins as the service  
> bootstrap file, like an int main(). I would probably *not* be into the  
> idea of passing sys.argv directly to a service, for example.

Why not?  It's just a list of strings.  You should be able to deal with it how you like.  (But as I said in a previous message: this is a separate issue.)

>> * It's very easy to accidentally make your plugin load your package  
>> for every other twistd daemon running out of the same plugin cache.
> 
> Never had this happen, but I think I can see where it might. It would  
> be solved by some kind of external metadata, though, right?

It already is solved by the external metadata... sort of.  If you look at the implementation of CachedPlugin, you can see that it actually already has a name and description!  There are two problems though: first is that this is hard-coded to be the module's name and docstring, but more importantly, there's just no way to get at those attributes via the getPlugin interface, which implicitly invokes '__conform__' via adaptation (and therefore load()).

> * The practice of putting a module under twisted/plugins/ 
>> $mymodulehere.py
>> upsets my equilibrium, the only reason I tolerate this kind of thing
>> is that I install my python code via rpms and I automatically get
>> installation, ownership and uninstallation done in a stable manner.
> 
> The only issue I've found with this is the issue of having to create  
> plugin cache files. Apart from the spurious error messages that were  
> (partially?) covered at the sprint recently, the benefit they provide  
> is definitely negated by bad permissions. I've done some acrobatics  
> inside my setup.py to make it work, but it involves different steps  
> depending on whether you're installing or building a package.

What is "this kind of thing", though?  The plugins have to go in some defined namespace in order to be enumerated.  Even if we were to implement something based on purely static metadata, you'd still have to list a directory to get at that metadata.  Making the namespace be owned by the module doing the importing makes sense.

For what it's worth, bzrlib does this too, but by convention in a bzr plugin you put _all_ your code into bzrlib/plugins/foo/*.py, and your package is bzrlib.plugins.foo.  (This will work fine with Twisted if you want to do it that way.)

> Still, it seems like most of their necessity would be negated by using  
> external metadata files. It seems to be the way most plugin systems  
> end up going in some way or another.

dropin.cache is an external metadata file ;-).

> Anyways, that's just my 2/100ths. I'm pretty happy with most of the  
> plugin model, but I think there's a lot of room for improvement,  
> particularly in the area of dropin.cache files.

Thanks for your feedback.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20110407/08426123/attachment.html>