[Twisted-Python] Multiple plugins in "twistd"

[exarkun at twistedmatrix.com]

On 12:07 pm, zadka.moshe at gmail.com wrote:
>Background
>
>Currently, "twistd" assumes one-run-one-plugin. It would be nice to 
>load up
>multiple plug-ins in Twisted, for many reasons. These include: serving 
>the
>same in-memory content via separate protocols, adding manhole to other
>plug-ins (so the end-deployer can add it to other things, as opposed to 
>the
>original implementor, and a catch-all category of "auxiliary services".
>
>Auxiliary services are those which are not useful in and of themselves, 
>but
>add value to a service which does something else of use.
>
>Examples of auxiliary services -- a logging service (that connects to 
>some
>logging protocol on start-up), a metrics service (that sends statistics 
>to
>a collector like statsd or riemann) or an error-sending service (to
>something like Sentry).
>
>Proposal
>
>tl;dr: four new tickets (codenamed SUBCOMMANDS, SERVICES, MANHOLE and
>PROVIDERS) and one old ticket (3538)
>
>SUBCOMMANDS: Add '+' as a special character in t.p.usage.Options. This
>behavior will be off by default, and controlled by an attribute on the
>Options instance "allowMultipleSubcommands".
>
>The attribute will only be checked when the first sub-command starts, 
>to
>allow setting it based on global flags. When the option is on, '-+' 
>will be
>passed as '+' to the Options instance, to allow sending plain '+' to
>sub-commands.

Having a new, weird, fragile syntax is probably the least interesting 
part of this.  I suggest not doing this part - or at the very least, not 
doing it first and not making it a general part of `Options`.

There are lots of other ways to get the service object from more than 
one service plugin.  For example, read lines from a file.  Or have a 
variation of `--python` or something else similar using the existing 
option syntax `Options` supports.

The more interesting part to get right is the underlying model which you 
discuss elsewhere.
>
>Glyph thinks there's a ticket for it. I couldn't find it in "search for
>tickets in 'core' whose description mentionds 'command'". Unless anyone 
>can
>find it, I'll open a new ticket.

I think there is a ticket for being able to use multiple twistd plugins. 
I don't think there's a ticket for a general change to `Options`.
>
>SERVICES (depends on SUBCOMMANDS): In twistd, set the flag aMS if
>'--allow-multiple-services' is given. Add to the application all 
>services.

If you skip the `SUBCOMMANDS` ticket described above, then you can skip 
this too.
>3538 (depends on SERVICES): If '--allow-multiple-services' is given, 
>and
>'--python <.tac file>' is given, process subcommands as usual.

Or just process the tac file and the subcommand if they're both given - 
without requiring an extra option?  The current behavior, "silently 
ignore one of the arguments", doesn't seem particularly worth keeping to 
me.
>PROVIDERS: Add a function,
>       "providersInHierarchy(IService, IInterface) -> List[IInterface]"
>that returns all services in the hierarchy which provide the interface.
>This ticket does not depend on any other tickets.

The first argument needs to be `IServiceCollection` instead of 
`IService`.
>MANHOLE (depends on PROVIDERS, SERVICES): Add a built-in twistd plugin
>named "manhole". The plugin will expose manhole as PB/telnet with a
>namespace that includes
>  {'services': providersInHierarchy(manholeService, IService)}
>This ticket technically could only depend on PROVIDERS, but to be 
>useful,
>it also depends on SERVICES

Manhole is part of Conch now and the telnet manhole is deprecated (and 
the PB manhole really should be deprecated - using a structured protocol 
for manhole isn't a bad idea but the existing implementation is half 
broken, mostly untested, exposes tons of implementation details as part 
of the public interface, etc.  If this were a piece of widely used 
software it would probably be worth gradually renovating - but it's 
basically used by no one so starting fresh makes more sense).

So it's part of Conch.  And ... it exists already.  I'm pretty sure no 
one will object if you add a new name to the default namespace.

Thanks for taking this on.

Jean-Paul