[Twisted-Python] Configuration, Persistance and Version Control

Matthew Walker matsaleh at austin.rr.com
Sat Jul 28 16:37:00 EDT 2001

Hmmm. Interesting context for the assignment Glyph just gave me - come up
with a friendly and robust config system for servers. I'm just coming up to
speed, so I have nothing to contribute yet, but I'm eager and this is a good
set of questions to begin with.

Sounds like one problem to solve is to differentiate between the initial
state with which you start/construct a server and the runtime state that
evolves during the use of the server, which is pickled. We need a robust and
easy way to a) establish the initial state (a la configure a server) and b)
enable that state to be saved and re-used in the future (e.g. kept in CVS or
other source control) while c) not mucking with the persistability of the
runtime state.

As I said, I am a twisted neophyte, so I'm going to spend much time learning
and listening and experimenting for a while first, but thanks for brining
the topic up.

-----Original Message-----
From: twisted-python-admin at twistedmatrix.com
[mailto:twisted-python-admin at twistedmatrix.com]On Behalf Of Kevin Turner
Sent: Saturday, July 28, 2001 2:26 PM
To: twisted-python at twistedmatrix.com
Subject: [Twisted-Python] Configuration, Persistance and Version Control

So, Tv and itamar were on channel today, asking about Twisted
philosophy, persistance, and configurability.  Lacking the
presence of anyone closer to the pure Twisted light, I tried
to be a good little minion and make the case...  But since
I'm the guy who doesn't quite believe in .spl files, I wasn't
able to be terribly convincing.

And now I could try to play his side of the argument with you guys,
but ya know, it'd be so much easier for me to just let his words do
the talking.  Scrollback excerpts:

<itamar|home> anyone who knows how twisted's config mechanism works?
<itamar|home> typically, glyph is on 24 hours a day unless I
          actually want to talk to a twisted developer
<Acapnotic> Wait, Twisted has a config mechanism?
<itamar|home> well, its server are persistent
<Acapnotic> right
<itamar|home> you coifgure them and then can pickle them to disk
<Acapnotic> right
<itamar|home> my question is: how do I configure them?
<itamar|home> can this mechanism be used with other software
<Acapnotic> You load them up, set them, and re-pickle them.  You
          can also set them through the interactive python interface on the
          telnet server, say
<Acapnotic> Now somehow, Glyph claims that this is easier than
          having configuration files.
<Acapnotic> I am not yet convinced of that, however.
<itamar|home> I am
<itamar|home> my software right now needs either something like
          that or an XML config file
<itamar|home> which'll end up being a pickle equivalent anyway, so
          there's no point


<Acapnotic> itamar: Well, so far as I've been able to tell, there's
          not overmuch magic there.  'mkfooserver' just creates an instance
          of a server object and pickle.dumps it, twistd just pickle.loads
          and invokes its the run() method.
<itamar|home> Acapnotic: well, my configs are pretty complex
<Tv> Acapnotic: But all that does is make your "config settings"
          into "command line settings".
<Tv> I don't see it making things very easy.
<Tv> And think about version upgrades.


<Tv> I see trouble with the approach; mostly losing the unix
          ubiquitousness (sp?). I can grep my config files; I store them in
<Tv> DavidC_: You want every program to contain some UI or command
          mechanism to fully configure it.
<Tv> There's a lot of things I'd wish someone would spell out about
          twisted. One thing is "how do I version control my configuration?"
<Tv> And don't give a trivial example..
<Tv> I have multi-hundred-line config files here.
<Tv> Shared between n hosts, via CVS.
<Tv> I'd love to hear more of the twisted philosophy..


<Tv> Personally, I have _needed_ programmability in config files
          many times.


<Tv> Acapnotic: Oh, I like version control. I wish I could put more
          things into it.
<Tv> Have you ever screwed up a config file?
<Tv> Have you ever downgraded something, and found out it won't
          understand the new config file, and had to recreate the config to
          match the older version?
<Tv> The same reasons you version control source code.

<Acapnotic> tv: you should post to the list at
          twisted-python at twistedmatrix.com
<Tv> Acapnotic: I'm not following any more lists until I have
          migrated my mail to another domain :)
<Acapnotic> tv: bah, you don't have to follow it, just post ;)


<Acapnotic> Tv: so if you want programmable config, is there a
          problem with just using python?  In which case your configuration
          process is a script that does "app = pickle.load(...) app['web'].
          port = 8080 ; app.append(ChickenServer()) ; ... ; pickle.dump(app,
          ...) ?
<Tv> Acapnotic: That may very well be what I'd do. But then the
          mechanism needs to support _both_ getting config programmatically
          and state from pickle, at the same time.
<Acapnotic> Well, if I follow this train, there is only the state
          of the application, since it is persistant.  That state may be
          changed or examined programmaticly.
<Tv> I see something beautiful there, but..
<Tv> noone has yet explained persistence to me well enough to
          convince me.
<Acapnotic> but what happens when you upgrade?  heh.
<Tv> What if a buggy program corrupts its state?
<Tv> Then you lose your config file also.
<Tv> There needs to be something else.
<Tv> Something that is less volatile.
<DavidC_> what do you mean? both config and state are the same
<Tv> No.
<Tv> config changes when *I* say it changes.
<itamar|home> nah, config is parsed and then "rendered" into state
<Tv> program can change it's state as it wants.
<Tv> Any bug in program must never change the port to bind to.
<Tv> :)
<Tv> I see config as sort of a factory for initial program states,
          if you want to put it that way.
<itamar|home> lets say by mistake someone does "del serverInstace.
<itamar|home> and then shuts down the server
<itamar|home> the pickled state is now wrong
<DavidC_> well that would be a bug.
<itamar|home> the point is that if you have an external, version
          controleld config file
<itamar|home> these bugs don't bite you, or even if they do at
          least they don't effect the configuration

Twisted-Python mailing list
Twisted-Python at twistedmatrix.com

More information about the Twisted-Python mailing list