[Twisted-Python] untwisting twistd

Tue Jun 23 11:27:09 EDT 2009

Thanks for the thoughts. I suppose I should have been a bit clearer on my specific use case that is not handled. Certainly, I do want to take advantage of twistd for the reasons listed below (daemonizing, handling log files, pidfile, etc); however, the problem that I have is that I have a very specific application that is going to be built into a product. There's no way that it could ever be installed or deployed elsewhere. Now, having said that I'd love to take advantage of the ApplicationRunner objects to fire up my application. The path to the application is baked in, the location for the pidfile and logfile should have sane defaults, and other various application parameters shouldn't change. In order to fire up the app with sane default, I should be able to issue one command with no arguments as the defaults for all of those should be baked into the system.

Sure, twistd provides default application values, but there's no easy way to override those values. There's no ServerOptions constructor function where I can configure the default values, or have some options removed from the list that appears if the user issues --help.

So, what I'm essentially arguing for is that although twistd is meant to solve a generic problem of generalizing server startups on various platforms for any number of applications, it's severely lacking in how I can take advantage of the ServerOptions to create one for my application. As it is now, I'm going to have to copy and paste the code and modify it as I want to create my own ServerOptions object that does have proper defaults. Right, now twisted.scripts._twistd_unis.ServerOptions inherits from app.ServerOptions, which is completely in appropriate as the UNIX ServerOptions have absolutely nothing to do with twistd itself. For example, I don't there to ever be a --file or --xml --source --report-profile --nothotshot --no_save or --savestats options. 
Really, this is a matter of presentation and locking down the application server options. I should be able to do something like this

ServerOptions( hide_items=('xml', 'file', 'source', 'report-profile'), =
		defaults={'logfile':'/var/log/foo.log', 'pidfile':'/var/run/foo.pid'}  )

To properly construct the subset of default twistd ServerOptions I need to have.

I guess in the mean time, I'll just write my own custom ServerOptions, which oddly, will mirror a bunch of code that already exists.

Thoughts?

-Jared

-----Original Message-----
From: twisted-python-bounces at twistedmatrix.com [mailto:twisted-python-bounces at twistedmatrix.com] On Behalf Of Alex Clemesha
Sent: Monday, June 22, 2009 2:36 PM
To: Twisted general discussion
Subject: Re: [Twisted-Python] untwisting twistd

Hi Jared,
First off, I'd like to say that I'm basically in agreement with exactly what
you are saying, and that is: "I should be able to write a Python
script that acts like twistd - not *be* twistd".

Phil does have some really good points, such as:
"""
The critics are usually not thinking about the real requirements of deploying a
configurable demonizing application in a production environment.
"""

And lastly, I'm glad to see that response from Glyph, as the methodology
he describes is basically how I understand one would create a
"personalized twistd".

In fact, I just got an email from a co-worker that wanted to know how to
either side-step "twistd" or write a custom version of it.  Here is my
response to him (I'd love some critic of my response, so I can provide
a better answer):

============= Email to co-worker: ===============

Yes, there is a better way of doing things.  This better way of doing things
is to write a "twisted plugin" and then, after that is working, using can
wrap calls to "twistd" with python's "os.system" calls, or shell script calls,
or whatever else you want.

I must admit, this whole procedure is pretty confusing (it was for me
at least) at first,
but let me try to explain why you might want to write a "twisted plugin" and
_then_ (possibly) wrap it with a shell script or a simple python script.
The main reasons are the following:

1) You _do_ want to use "twistd", because it does lots of useful
things like daemonizing, handling logfiles and pidfiles, etc.
2) You also want to pass config code to the app you are starting up
(via "twistd") - optimally from the command line and a settings file.

Now, once the above is in place, you then can write some wrapper code
on top of the "twistd" calls for convenience for users.

Here is the "plugin" documentation:
http://twistedmatrix.com/projects/core/documentation/howto/plugin.html

But I've found that it is easier to go from an example, so try typing this:

twistd -n web --path="."

Now open up firefox to http://localhost:8080
This starts up a twisted web server, serving files from the local directory.
This is a twisted web plugin in action, and it is implemented with
these 2 files:

http://twistedmatrix.com/trac/browser/trunk/twisted/plugins/twisted_web.py
http://twistedmatrix.com/trac/browser/trunk/twisted/web/tap.py

============ end email to co-worker ==============

So basically I've found that the best thing to do is to extend
"twistd" via writing a plugin, and then you can write all kinds
of simple wrappers on top of it to provide a more user-friendly
and a your-app-specific command line tool.  Thoughts from Twisted gurus?

thanks,
Alex

On Mon, Jun 22, 2009 at 1:30 PM, Jared Gisin<jared.gisin at isilon.com> wrote:
> I am currently working on writing an application using the twisted
> framework. The problem that I have is that I do not want to use twistd. In
> fact, I find twistd to be inappropriate as it defaults pidfile and logfile
> to something specific to twisted, not my application, and it seem completely
> silly to have to pass in -pidfile and -logfile params when stating it up.
> Those parameters should be specified as part of the Application object
> itself, and should allow for CLI override. Further, as I want my daemon that
> I'm writing to look and act like a daemon (UNIX), I do not want to be
> invoking some general "twistd" command and passing a ton of options.
> Ideally, I should be able to write a Pythons script that acts like twistd,
> but can bake in default options and hide/expose other ServerOptions as I so
> see fit.
>
>
>
> For example, if I run,
>
>
>
> my_daemon
>
>
>
> It should fire up the daemon
>
>
>
> my_daemon -help
>
>
>
> should dump the help
>
>
>
> my_daemon -pidfile=/var/run/my_daemon.pid
>
> my_daemon stop
>
> my_daemon restart
>
>
>
> should all function accordingly. I want to avoid all of this "twistd -y"
> business, as the daemon, isn't "twistd" it's my code. I should be able to
> use the ApplicationRunner, UnixApplicationRunner, etc "runners",  however,
> as they are currently written, they are twistd specific. For example, in
> 8.2, the def checkPid function bakes in, "            sys.exit("""\
>
> Another twistd server is running, PID %s\n
>
> This could either be a previously started instance of your application or a
>
> different application entirely. To start a new one, either run it in some
> other
>
> directory, or use the --pidfile and --logfile parameters to avoid clashes.
>
> """ %  pid)"
>
>
>
> Which is iditic as my application may not even expose a -pidfile or a
> -logfile and it's definatley not called twistd. The platform specific
> application runners need to be more generalized so users can write their own
> twistd equivalent to include only what they want.
>
>
>
> Perhaps I'm missing some key point of twisted that already does this. If so,
> I'd love to be guided in the right direction. If not, and I've got a valid
> point here, I'd love to know how I can help to remedy this problem. I think
> it's great that twistd provides application management classes, but they are
> unnecessarily intertwined with twistd which should be completely optional.
>
>
>
> -Jared
>
>
>
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>

-- 
Alex Clemesha
clemesha.org

_______________________________________________
Twisted-Python mailing list
Twisted-Python at twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python