[Twisted-Python] Twisted Euphrates (Modified by Glyph Lefkowitz)

[Glyph Lefkowitz]

Unfortunately, I've been out of the loop on Twisted development for 
some time.  During the course of the PyCon sprints, I've been catching 
up with what's been going on and trying to do some planning for future 
directions in development.

Yesterday Itamar and I had our quarterly screaming argument about 
obscure details, and it didn't come to blows, which I believe means 
that spring is right around the corner!

The major issue we've been discussing is splitting up Twisted into 
multiple packages.  Twisted has been encountering some growing pains, 
due to a mismatch between the way it's developed and the way it's 
distributed.  We (T.M.Labs) are really a community of developers 
working on several interrelated projects with different goals, 
stability, and release cycles, but we are distributing one monolithic 
system, with code of vastly different qualities in the same tarball and 
sometimes even in the same package.

The primary problem that this leads to is revision coupling.  This is 
probably old news to anyone trying to deploy a large system which 
relies on any part of Twisted besides the base reactor (and maybe 
twisted.web).  Scenarios like this happen all the time:

	- Twisted v.N.M.0 is released.
	- Project P uses twisted components X and Y.  (Generally X and Y are 
marked 'Unstable' or 'Semi-Stable', but are very useful or heavily 
marketed or otherwise convince unsuspecting users to trust their 
stability.)
	- Project Q uses twisted components Y and Z.
	- Project P v. 1 is released.
	- Project Q v. 1 is released.
	- Project P gives useful feedback to twisted developer A who then 
fixes several bugs in component X which are critical to the functioning 
of the as-yet-unreleased P v.2
	- Twisted developer B working on project Q commits substantial 
"improvements" to component Y, which change the functionality in a way 
that breaks project X.
	- Radix prepares an alpha/bugfix release, v.N.M.1

At this point, Project Q is perfectly happy, because the most recent 
alpha works for their code.  However, Project P is very unhappy, 
because there is no release which includes their bugfix but does NOT 
include code which forces them to upgrade some functionality in an a 
way which is incompatible with the previous micro-version release of 
Twisted which their users presumably already have installed.  The 
release notes have to be very specific, and because nobody reads them 
anyway, a lot of confusion results.

While there are problems with splitting Twisted into multiple projects, 
I think that the time has come to do so.  Originally the feedback I 
heard from most users was that it was great to have lots of different 
packages available so readily; the "batteries included" philosophy does 
have a lot going for it.  Now, most of what I'm hearing is complaints 
about the above scenario, not the least of which from my own employer 
:-).  We are more often in the role of project Q than project P 
(although we have been in both more than once) and I'd prefer not to 
have a few projects who employ active Twisted developers control the 
release cycle based on their requirements.

Because splitting the project involves moving some code around, we 
should take the opportunity to clean up another problem: our code has 
no effective separation between interface and implementation.  Within 
the project this is fine, because we can easily refactor any uses of 
deprecated APIs.

While we aren't going to break the existing package-structure 
interfaces (or even deprecate them, for the time being), this means 
that in the future, imports for twisted projects will probably look 
like:

	from twisted_core import reactor
	from twisted_web import Resource

The way this will be implemented, a top-level "twisted_core" module (or 
possibly, package, but I would like as little nesting as possible here) 
will import explicitly public names from a _twisted_core implementation 
package.  For the forseeable future, the twisted.* package hierarchy 
will continue to work for imports; however, considering how informally 
specified this interface has been, I imagine improved documentation 
will emerge for the newer interface packages.

Personally I wanted to call these modules "twisted_web_1", and I also 
wanted to talk about versioning here, but amidst strenuous objections 
from Anthony Baxter and Itamar ST I will refrain :).

Suffice it to say that I believe that regardless of many potential 
strategies available for versioning, a separate module for interface 
vs. implementation will make it possible to determine what is public 
and usable from a deployed application using the Python interactive 
prompt or reading the code.  Since many packages don't have stability 
declarations anyway, I think this will result in many fewer 
unintentional uses of private code.

The first package that we are going to try this on is twisted.web, 
because there is a major rewrite pending integration of all the fixes 
and redesign that has happened in the Nevow package.  If this goes 
well, other packages will use the same naming conventions.  In any 
case, we will be migrating packages out of the Twisted core. for 
example: lore, conch, manhole, and mail.

As part of this process we hope to improve the website to reflect the 
name "Twisted" as primarily a part of "Twisted Matrix Labs", the group 
of developers working on these projects.  Similar to tigris.org, TML is 
"a mid-sized open source community focused on building better tools for 
networked application development".  Package maintainers: each project 
should have its own sub-site, so consider a project description and any 
other documentation you might want on that site.

As far as our development process, we will be keeping all the code in 
one subversion repository and still running BuildBot over the entire 
system in a way similar to how we're doing it now.  There are lots of 
possible improvements to the process, all the way up to an independent 
package management system using distutils: please, don't recommend any 
of these.  We don't want to make a huge number of changes at the same 
time, and while this moving / renaming is fairly mechanical, it will 
touch almost every file in the world; we need to make sure it all works 
before we begin adding features based on it.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
URL: </pipermail/twisted-python/attachments/20040323/b5ebd1cf/attachment.sig>