[Twisted-Python] Twisted Plugins - Implementation Discussion

[Glyph Lefkowitz]

On Apr 7, 2011, at 9:19 AM, Andrew Bennetts wrote:

> Itamar Turner-Trauring wrote:
> […]
>> So, the design has to *not* rely on caching working.
> 
> FWIW: this is an achievable goal.  I have 32 different bzr plugins
> currently installed, and here's the difference they make:
> 
>   $ time bzr --no-plugins rocks
>   It sure does!
> 
>   real	0m0.075s
> 
>   $ time bzr rocks
>   It sure does!
> 
>   real	0m0.119s
> 
> So that's about 1.5ms per plugin, on average.  With a hot disk cache, at
> least…

Is your cache as hot for Twisted as for bzr?  Have you replicated these results in a randomized, double-blind clinical trial? ;-)

I'm not surprised that bzr has faster startup though; twistd has not been (and doubtful will ever be) nearly so ruthlessly optimized.  Maybe it's time to put a startup benchmark on <http://speed.twistedmatrix.com/>, at least that way we could keep track.

> For comparison, 'twistd --version' takes 116ms, with a dropin.cache and
> (I think, although how can I tell?) no plugins installed.

Twisted itself installs 22 dropins (python files which each define at least one plugin), which comprise 48 plugins of various types, so there are always some.  You should be able to tell, though.  It's pathetic that we don't have a command-line tool to inspect the available plugins and what they're doing.  Independent of the other issues under discussion here: <http://twistedmatrix.com/trac/ticket/5039>.

But this is all moot.  'twistd --version' doesn't scan for plugins, so that's all just the normal startup time; apparently we import too much in the first place.  The thing to compare with is 'twistd --help' or even just 'twistd [some-plugin]' (since invoking one plugin actually loads all of them).  Plus - this is really the genesis for this thread - the dropin.cache isn't really saving us much work at all right now, because all the plugins get loaded anyway for all practical uses of plugin scanning.

> In part, we achieve this via the bzrlib.lazy_import hack, which plugins
> can and often do use, and by encouraging plugin authors to put as little
> code into their __init__.py files as possible.  A typical plugin's
> __init__ might do just:
> 
>    # This is example_plugin/__init__.py
>    # The actual command implementation is in
>    # example_plugin/example_commands.py
>    from bzrlib import commands
>    commands.plugin_cmds.register_lazy('cmd_class_name', [],
>        'bzrlib.plugins.example_plugin.example_commands')

This looks very similar to ServiceMaker.

> Glyph's expressed scepticism that plugin authors and maintainers will
> know to keep their __init__.py files cheap to import.  Bazaar's
> experience is different.  Partly that's probably because the Bazaar
> community has paid a fair bit of attention to start up time and I
> suppose Twisted doesn't have that.

Yeah, bzr's audience makes this easier.  For one thing, the audience is much bigger :), but more importantly, bzr is a user-facing tool which users are running _constantly_ at the command line.  The only visible consequence of a rogue twistd plugin is that your server which runs for days at a time takes 0.2s longer to start; the real problem sets in later, where your 25 subprocesses are suddenly consuming an additional 50meg each because of the extra plugin they loaded.  You do find this eventually, it's just rare to find it while you're writing the plugin.

> But I think also it's partly because
> we've provided tools to help people diagnose what/who to blame for bzr
> being slow to start, like 'bzr --profile-imports', and even the crude
> 'time bzr rocks'.


Yes.  These are a great idea, and there's no excuse that Twisted's plugin system is so difficult to inspect and debug.  A couple of good tools would address a wide range of plugin issues, many of them much more interesting than performance, like the ever-popular "why isn't my plugin getting loaded".  Thanks for the impetus to file the ticket above.  (I kinda hope it's a dup, but I couldn't find one.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20110407/264d4da8/attachment.htm