It'd be nice to be able to use a .tac file and a plugin at the same time

[radix]

.tacs and plugins have complementary features:

• tac files specify and configure the application;
• plugins specify and configure services which will be added to the application.

Therefore, I propose that 'twistd -y conf.tac web' should function such that 'conf.tac' be used to create and configure the application while the service that the 'web' plugin returns be added to it.

Right now, it seems that such a command line simply ignores the 'conf.tac' file, and does not evaluate it at all.

[msteder]

Apply with "patch -p1 < tac_plus_plugins_3538.patch"

[msteder]

I'm hoping to be able to customize logging while using the twisted plugin mechanism and this seemed like a relatively painless way to do it.

I started looking into this and sure enough, some quick tweaks to twisted.application.app and it appears to work. However, the Application constructed in the specified *.tac file does not get passed any of the configuration parameters passed to either to twistd or your plugin. Having to edit your tac file or use a different tac for changes to your applications logging seems like the wrong solution.

Unfortunately it looks like there is no way to pass the configuration into the tac file. By the time the tac file is imported the application is already instantiated and it's too late to customize any configuration.

For example, I have the following tac just for testing:

# dailylog.tac
from twisted.application.service import Application
from twisted.python.log import ILogObserver, FileLogObserver
from twisted.python.logfile import DailyLogFile

application = Application("myapp")
logfile = DailyLogFile("my.log", "/tmp")
application.setComponent(ILogObserver, FileLogObserver(logfile).emit)

So now with the tac + plugin scheme one can do something like:

$ twistd -y dailylog.tac web --path=~/public_html

I'm thinking that the above may be useful for some use cases but to support the kind of logging configuration tweaks I'd like to do modified twistd to use a new interface: IApplicationMaker to create an application from the plugin ServiceMaker. There's an adapter for existing ServiceMaker's to create a default Application like twistd currently does.

So if you need to be able to access twistd configuration parameters you can simply modify your plugin ServiceMaker to implement IApplicationMaker and implement the single method, makeApplication.

And the plugin ServiceMaker would be modified to include:

implements(IPlugin, IServiceMaker, IApplicationMaker)
...
def makeApplication(self, options):
    application = service.Application(options.subCommand)
    logFile = DailyLogFile(options.logfile, "/tmp")
    application.setComponent(ILogObserver, FilelogObserver(logfile).emit)
    return application

So an example, if one implemented IApplicationMaker for the Twisted Web plugin:

$ twistd --logfile=/home/me/web.log web --path=~/public_html

I've included in the patch a custom logging example on top of twisted web called 'logexample'.

[exarkun]

Unfortunately it looks like there is no way to pass the configuration into the tac file.

Quite true. "tac" stands for "Twisted Application Configuration". Passing configuration to your configuration doesn't make much sense.

By the time the tac file is imported the application is already instantiated and it's too late to customize any configuration.

I don't really understand this. The application is defined in the tac file. So I don't see how it could be the case that the application is already instantiated by the time the tac file is evaluated.

I'm thinking that the above may be useful for some use cases but to support the kind of logging configuration tweaks I'd like to do modified twistd to use a new interface: IApplicationMaker to create an application from the plugin ServiceMaker.

I'm somewhat skeptical of this direction. One goal of twistd plugins is to be able to run several of them at once. This isn't current possible. However, if it becomes possible for each twistd plugin to define its own Application, then it'll probably never be possible.

Meanwhile, as a solution to the problem of command line configuration of the Application parts of things, this approach has some shortcomings: it will require every twistd plugin to repeat the same implementation of support for common options, it will allow supported options to diverge amongst different twistd plugins, it will foist cross-platform issues onto application code where it doesn't really belong.

I'm very happy that you're interested in this, since it's quite a rough part of Twisted that a lot of people would like to see fixed. My suggestion is to be even more creative in trying to find a solution. The current form of twistd is the result of years of accretion with little thought for future extensibility. Loading a .tac file and a twistd plugin in the same command line is an example of a pragmatic solution which adds to this accretion. We should probably do it because it is a significant increase in usability for twistd. A good solution, however, will probably look wildly different, and will may not fit nicely into the current structure of twistd. That's okay. However, we shouldn't block the immediate simple fix on inventing the superior fix.

[msteder]

Replying to exarkun:

I'm somewhat skeptical of this direction. One goal of twistd plugins is to be able to run several of them at once. This isn't current possible. However, if it becomes possible for each twistd plugin to define its own Application, then it'll probably never be possible. Meanwhile, as a solution to the problem of command line configuration of the Application parts of things, this approach has some shortcomings: it will require every twistd plugin to repeat the same implementation of support for common options, it will allow supported options to diverge amongst different twistd plugins, it will foist cross-platform issues onto application code where it doesn't really belong. I'm very happy that you're interested in this, since it's quite a rough part of Twisted that a lot of people would like to see fixed. My suggestion is to be even more creative in trying to find a solution. The current form of twistd is the result of years of accretion with little thought for future extensibility. Loading a .tac file and a twistd plugin in the same command line is an example of a pragmatic solution which adds to this accretion. We should probably do it because it is a significant increase in usability for twistd. A good solution, however, will probably look wildly different, and will may not fit nicely into the current structure of twistd. That's okay. However, we shouldn't block the immediate simple fix on inventing the superior fix.

I have a patch that just includes the tac + plugin stuff that I will attach later today when I get home.

I'm thinking perhaps a way to address both my need for application configuration through the command line and your concerns about flexibility and maintainability is to allow plugins to specify a components they would like to override on the Application. This way, the Application could be instantiated by twistd, using either the default Application or one in a specified tac file, and the the plugins which already handle twistd arguments could specify a ILogObserver component and twistd could apply it to the application.

This way no plugin creates an application but any plugin can modify the application by specifying components that should be added, set, unset, or removed.

To address your concern about multiple plugins all having logic for customizing logging I could modify things so that multiple plugins can be used together. Then my example logging plugin really could become a real optional plugin for use in multiplugin scenarios. This way all the code for customizing logging based on commandline arguments to twistd could be included in the logging plugin and doesn't end up being duplicated all over unless someone really wants to customize it or has a use case that isn't supported.

For example, you might do something like:

$ twistd -y myapp.tac logging_plugin --logfile=my.log --rotate=daily --backend=pythonlogging web --path=~/public_html

Is there a ticket for supporting multiple plugins?