[Twisted-Python] Twisted Euphrates (Modified by Glyph Lefkowitz)
glyph at divmod.com
Tue Mar 23 14:00:21 EST 2004
Unfortunately, I've been out of the loop on Twisted development for
some time. During the course of the PyCon sprints, I've been catching
up with what's been going on and trying to do some planning for future
directions in development.
Yesterday Itamar and I had our quarterly screaming argument about
obscure details, and it didn't come to blows, which I believe means
that spring is right around the corner!
The major issue we've been discussing is splitting up Twisted into
multiple packages. Twisted has been encountering some growing pains,
due to a mismatch between the way it's developed and the way it's
distributed. We (T.M.Labs) are really a community of developers
working on several interrelated projects with different goals,
stability, and release cycles, but we are distributing one monolithic
system, with code of vastly different qualities in the same tarball and
sometimes even in the same package.
The primary problem that this leads to is revision coupling. This is
probably old news to anyone trying to deploy a large system which
relies on any part of Twisted besides the base reactor (and maybe
twisted.web). Scenarios like this happen all the time:
- Twisted v.N.M.0 is released.
- Project P uses twisted components X and Y. (Generally X and Y are
marked 'Unstable' or 'Semi-Stable', but are very useful or heavily
marketed or otherwise convince unsuspecting users to trust their
- Project Q uses twisted components Y and Z.
- Project P v. 1 is released.
- Project Q v. 1 is released.
- Project P gives useful feedback to twisted developer A who then
fixes several bugs in component X which are critical to the functioning
of the as-yet-unreleased P v.2
- Twisted developer B working on project Q commits substantial
"improvements" to component Y, which change the functionality in a way
that breaks project X.
- Radix prepares an alpha/bugfix release, v.N.M.1
At this point, Project Q is perfectly happy, because the most recent
alpha works for their code. However, Project P is very unhappy,
because there is no release which includes their bugfix but does NOT
include code which forces them to upgrade some functionality in an a
way which is incompatible with the previous micro-version release of
Twisted which their users presumably already have installed. The
release notes have to be very specific, and because nobody reads them
anyway, a lot of confusion results.
While there are problems with splitting Twisted into multiple projects,
I think that the time has come to do so. Originally the feedback I
heard from most users was that it was great to have lots of different
packages available so readily; the "batteries included" philosophy does
have a lot going for it. Now, most of what I'm hearing is complaints
about the above scenario, not the least of which from my own employer
:-). We are more often in the role of project Q than project P
(although we have been in both more than once) and I'd prefer not to
have a few projects who employ active Twisted developers control the
release cycle based on their requirements.
Because splitting the project involves moving some code around, we
should take the opportunity to clean up another problem: our code has
no effective separation between interface and implementation. Within
the project this is fine, because we can easily refactor any uses of
While we aren't going to break the existing package-structure
interfaces (or even deprecate them, for the time being), this means
that in the future, imports for twisted projects will probably look
from twisted_core import reactor
from twisted_web import Resource
The way this will be implemented, a top-level "twisted_core" module (or
possibly, package, but I would like as little nesting as possible here)
will import explicitly public names from a _twisted_core implementation
package. For the forseeable future, the twisted.* package hierarchy
will continue to work for imports; however, considering how informally
specified this interface has been, I imagine improved documentation
will emerge for the newer interface packages.
Personally I wanted to call these modules "twisted_web_1", and I also
wanted to talk about versioning here, but amidst strenuous objections
from Anthony Baxter and Itamar ST I will refrain :).
Suffice it to say that I believe that regardless of many potential
strategies available for versioning, a separate module for interface
vs. implementation will make it possible to determine what is public
and usable from a deployed application using the Python interactive
prompt or reading the code. Since many packages don't have stability
declarations anyway, I think this will result in many fewer
unintentional uses of private code.
The first package that we are going to try this on is twisted.web,
because there is a major rewrite pending integration of all the fixes
and redesign that has happened in the Nevow package. If this goes
well, other packages will use the same naming conventions. In any
case, we will be migrating packages out of the Twisted core. for
example: lore, conch, manhole, and mail.
As part of this process we hope to improve the website to reflect the
name "Twisted" as primarily a part of "Twisted Matrix Labs", the group
of developers working on these projects. Similar to tigris.org, TML is
"a mid-sized open source community focused on building better tools for
networked application development". Package maintainers: each project
should have its own sub-site, so consider a project description and any
other documentation you might want on that site.
As far as our development process, we will be keeping all the code in
one subversion repository and still running BuildBot over the entire
system in a way similar to how we're doing it now. There are lots of
possible improvements to the process, all the way up to an independent
package management system using distutils: please, don't recommend any
of these. We don't want to make a huge number of changes at the same
time, and while this moving / renaming is fairly mechanical, it will
touch almost every file in the world; we need to make sure it all works
before we begin adding features based on it.
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://twistedmatrix.com/pipermail/twisted-python/attachments/20040323/b5ebd1cf/attachment.pgp
More information about the Twisted-Python