[Twisted-Python] Multiple plugins in "twistd"

[exarkun at twistedmatrix.com]

On 07:29 pm, glyph at twistedmatrix.com wrote:
>
>>On May 20, 2015, at 04:13, exarkun at twistedmatrix.com wrote:
>>
>>On 19 May, 09:01 pm, tom.prince at ualberta.net wrote:
>>>Glyph <glyph at twistedmatrix.com> writes:
>>>>You could also find some other way to split the argument list but 
>>>>"+" doesn't seem particularly obscure in this context to me.  (If 
>>>>there's really a need to pass a literal "+" to a plugin we could add 
>>>>an escaping syntax as well.)
>>>
>>>I think if we are adding syntax, then we should also add escaping at 
>>>the
>>>same time.
>>>
>>>On a related note, when designing this kind of syntax, I think it is
>>>often valuable to explictly leave some of the space as an explict 
>>>error,
>>>to leave freedom to extend the syntax in the future.
>>
>>I think this is 100% correct.  This is part of why I don't want the 
>>syntax added to `Options`.  If it's a feature of, say, a "compose" 
>>twistd plugin then you can always throw the whole "compose" twistd 
>>plugin in the trash and start again.  That gives you quite a lot of 
>>space for syntax changes. :)
>>
>>(And of course, not introducing a syntax at all leaves you even more 
>>room... but talking this crowd out of inventing weird syntaxes is 
>>probably an exercise in futility.)
>
>Much as I love weird syntaxes, using "+" as a separator for a 'compose' 
>plugin seems about as straightforward as I could imagine such a thing 
>working.  Do you have an alternate proposal that is less 'weird'?  I 
>wouldn't promise to accept it but I wouldn't want to go with something 
>unnecessarily weird just due to a failure of my imagination.

It's possible this is a case of necessary weirdness - at least, if 
you're dead-set on some kind of option-based CLI interface to this 
functionality.

Think about other CLI tools that try to offer this kind of composition. 
Only two really come to mind for me and one of those is not exactly well 
regarded for its novel syntax (hint - it rhymes with "schmestreamer).

This is not to say that this kind of composition is a bad thing - or 
even that no other widely-used software supports the kind of invoke- 
several-things-of-stuff behavior that we're talking about providing 
access to here.

However, I do think it is the case that most of the world tries to solve 
this problem without trying to force everything into a world-view framed 
by GNU getopt().

With that in mind, here are some alternatives:

  (0) The null proposal

    $ twistd compose \
        web --port 80 +
        manhole --auth foo + \
        logging --path /var/run

  (1) Borrow `--` which already means "stop parsing options here".  It 
makes sense if you think about sub-commands as existing on a stack and 
`--` meaning "pop the stack".  Possibly not actually feasible since I 
think `Options` already knows what `--` means (by virtue of actually 
*using* getopt()) and uses it to separate options from positional 
arguments (and therefore it's not an unambiguous separator - but I 
haven't double-checked this).

    $ twistd compose \
        web --port 80 -- \
        manhole --auth foo -- \
        logging --path /var/run

  (2) Put things into strings which already have well-defined escape 
syntax:

    $ twistd compose \
        'web --port 80' \
        'manhole --auth foo' \
        'logging --path /var/run'

  (3) Load things from a file (really just a variation on (2)):

    $ cat myapp.compose
    web --port 80
    manhole --auth foo
    logging --path /var/run
    $ twistd compose myapp.compose

Note these aren't really syntax ideas.  Apart from (1), they're ideas 
for not needing a new syntax.  As soon as you actually invent a new 
syntax you invite comments like "Well, maybe you should use `;` instead 
of `+` because `;` is already kind of ingrained as a separator and 
people are used to having to escape it" (which, see, I put in quotes, 
because I'm totally not saying that, even if it is totally true) to 
which you can only really reply that your gut agrees or your gut 
disagrees and it's always a bummer when the best support you have for 
some programming decision is some gut feeling you had.

I might actually be somewhat attracted to (3) because:

  * I don't really want to build up giant piles of configuration on my 
command line and then rely on my shell history for being able to find it 
again.  myapp.compose is something I could even put in my version 
control system.

  * Despite shell quoting and escaping rules being perfectly well 
defined, I'd really much rather not have to deal with an automatic extra 
layer of quotes.

And a last point to consider about new syntaxes - what are the security 
considerations of a command like:

    $ twistd compose \
        web --port ${WEB_PORT} +
        manhole --auth foo + \
        logging --path /var/run

I'll grant there's a bug there (the shell variable should be quoted - 
but I only know one person who consistently gets that right).

Regardless, I think any proposal would greatly benefit from a much 
greater consideration of use-cases and examples than I've yet seen 
discussed in this thread.

Jean-Paul