[Twisted-Python] Multiple plugins in "twistd"

[Glyph]

> On May 19, 2015, at 05:59, exarkun at twistedmatrix.com wrote:
> 
> On 12:07 pm, zadka.moshe at gmail.com wrote:
>> Background
>> 
>> Currently, "twistd" assumes one-run-one-plugin. It would be nice to load up
>> multiple plug-ins in Twisted, for many reasons. These include: serving the
>> same in-memory content via separate protocols, adding manhole to other
>> plug-ins (so the end-deployer can add it to other things, as opposed to the
>> original implementor, and a catch-all category of "auxiliary services".
>> 
>> Auxiliary services are those which are not useful in and of themselves, but
>> add value to a service which does something else of use.
>> 
>> Examples of auxiliary services -- a logging service (that connects to some
>> logging protocol on start-up), a metrics service (that sends statistics to
>> a collector like statsd or riemann) or an error-sending service (to
>> something like Sentry).
>> 
>> Proposal
>> 
>> tl;dr: four new tickets (codenamed SUBCOMMANDS, SERVICES, MANHOLE and
>> PROVIDERS) and one old ticket (3538)
>> 
>> SUBCOMMANDS: Add '+' as a special character in t.p.usage.Options. This
>> behavior will be off by default, and controlled by an attribute on the
>> Options instance "allowMultipleSubcommands".
>> 
>> The attribute will only be checked when the first sub-command starts, to
>> allow setting it based on global flags. When the option is on, '-+' will be
>> passed as '+' to the Options instance, to allow sending plain '+' to
>> sub-commands.
> 
> Having a new, weird, fragile syntax is probably the least interesting part of this.  I suggest not doing this part - or at the very least, not doing it first and not making it a general part of `Options`.
> 
> There are lots of other ways to get the service object from more than one service plugin.  For example, read lines from a file.  Or have a variation of `--python` or something else similar using the existing option syntax `Options` supports.
> 
> The more interesting part to get right is the underlying model which you discuss elsewhere.

I love new, weird syntaxes, but I have to agree here.  Let's do it with a boring syntax with no compatibility implications first.  The boring syntax would actually be quite similar to the exciting one; a "compose" plugin.  Consider: "twistd compose web --path . + words --irc-port=6789".  There's no need to make this a general feature of Options to do it this way; just have the "compose" plugin take its argument list and pass it to two discrete other Options instances as lists of strings, looking for a "+" simply as the place to split the argument list.  You could also find some other way to split the argument list but "+" doesn't seem particularly obscure in this context to me.  (If there's really a need to pass a literal "+" to a plugin we could add an escaping syntax as well.)

>> Glyph thinks there's a ticket for it. I couldn't find it in "search for
>> tickets in 'core' whose description mentionds 'command'". Unless anyone can
>> find it, I'll open a new ticket.
> 
> I think there is a ticket for being able to use multiple twistd plugins. I don't think there's a ticket for a general change to `Options`.

I was referring to the multiple-twistd-plugin ticket.

>> SERVICES (depends on SUBCOMMANDS): In twistd, set the flag aMS if
>> '--allow-multiple-services' is given. Add to the application all services.
> 
> If you skip the `SUBCOMMANDS` ticket described above, then you can skip this too.
>> 3538 (depends on SERVICES): If '--allow-multiple-services' is given, and
>> '--python <.tac file>' is given, process subcommands as usual.
> 
> Or just process the tac file and the subcommand if they're both given - without requiring an extra option?  The current behavior, "silently ignore one of the arguments", doesn't seem particularly worth keeping to me.

It seems halfway plausible to me that someone could have some automation that chooses a plugin and/or a python file and expects that one or the other will be ignored.  So we should do the usual incompatibility-notification dance but I don't anticipate any objections.

>> PROVIDERS: Add a function,
>>      "providersInHierarchy(IService, IInterface) -> List[IInterface]"
>> that returns all services in the hierarchy which provide the interface.
>> This ticket does not depend on any other tickets.
> 
> The first argument needs to be `IServiceCollection` instead of `IService`.
>> MANHOLE (depends on PROVIDERS, SERVICES): Add a built-in twistd plugin
>> named "manhole". The plugin will expose manhole as PB/telnet with a
>> namespace that includes
>> {'services': providersInHierarchy(manholeService, IService)}
>> This ticket technically could only depend on PROVIDERS, but to be useful,
>> it also depends on SERVICES
> 
> Manhole is part of Conch now and the telnet manhole is deprecated (and the PB manhole really should be deprecated - using a structured protocol for manhole isn't a bad idea but the existing implementation is half broken, mostly untested, exposes tons of implementation details as part of the public interface, etc.  If this were a piece of widely used software it would probably be worth gradually renovating - but it's basically used by no one so starting fresh makes more sense).
> 
> So it's part of Conch.  And ... it exists already.  I'm pretty sure no one will object if you add a new name to the default namespace.
> 
> Thanks for taking this on.

Yeah.  It's great to have you around again, Moshe :).

-glyph