Split Twisted into many packages and release 2.0

[itamarst]

This involves:
1. Naming of package
2. Moving code, docs, examples
3. New section on website
4. New version number
What else?

[z3p]

1. I vote for just naming it Conch.
3. http://www.twistedmatrix.com/users/z3p/conch/

[itamarst]

Just Conch sounds ok to me.
We need to put the website on the main tm.com bit (tm.com/products/conch?).
License? Copyright holder?

[glyph]

On Mar 30, 2004, at 10:01 AM, Itamar Shtull-Trauring [[Twisted issue
tracker]] wrote:
> License? Copyright holder?
Paul, do you want me to continue to hold the copyright?  I can do so if
you want to declaim responsibility, but it's really your code, so you
can feel free to take it... your decision.
I'd recommend BSD license, personally, since we should be transitioning
all of Twisted to it as we break it up.

[z3p]

I'll take the copyright (since I'm now old enough to hold it :).
BSD is good, and it seems to work for OpenSSH just fine.
Also:
5. Transitioning the Debian package to something that makes it more obvious that
it's a client/server, not just a library.

[radix]

I'm hijacking this issue to be the general "twisted split" issue.
This is the *temporary* plan, to get the release schedules broken up as simply
as possible (from glyph's messege to the list, "The Argument I Lost", brought up
to date). Discussion of what happens "eventually" will take place somewhere
other than this bug.
 * The repository mostly stays how it is.
 * Buildbot keeps running against trunk, but against a series of application
   packages instead of twisted.test (done: It's doing "trial -R twisted" which
   catches stuff like twisted.mail.test)
 * Package names do not change.
Eventually, one or all of these may need to change.  However, all that will be
happening to split out packages is:
 * Modules in protocols/, test/, and tap/ move to an appropriate top-level
   package, with backward-compatibility wrapper.
 * The release script will be changed to generate separate tarballs for various
   top-level packages.
 * We will create a topfiles/ directory or somesuch to contain READMEs,
   setup.py's, etc., for each subproject.
In addition, both bin/ and doc/ will have project-specific subdirectories.
So far I have done work to separate news, mail, and names. The repository
rearrangement is pretty easy, the hard part is going to be the release
automation, which still is not entirely planned.

[radix]

Oh, right. I should mention what the new packages will be:
* twisted.mail += all mail protocols (done)
* twisted.news += nntp               (done)
* twisted.names += dns               (done)
* twisted.words += twisted.im, irc, msn, oscar, toc, jabber
* twisted.web += http
* twisted.flow
* twisted.conch += ssh
* twisted.lore
* twisted.pair
* twisted.runner
* twisted.xish
application, cred, internet, persisted, protocols (if there are any left),
python, scripts, will all *definitely* be in twisted core.
enterprise, spread, manhole, trial *probably* will be in twisted core.

[radix]

flow, pair, and lore are reorganized. yet to do:
 * runner - crap, need to move runner.procutils somewhere else, perhaps
twisted.python.procutils
 * xish - maybe this will just be with twisted.jabber or something
 * conch
 * web

[radix]

runner's done
words's done
horg my head is going to explode

[radix]

conch + web are done
only reorg left is jabber/xish stuff. what're we doing with that?
twisted.jabber? move jabber to twisted.words? what to do with xish?
i just want to get it over with someone tell me what to do

[radix]

jabber is in words
xish is where it is because itamar said just leave it
time for DOC/!!!

[radix]

doc is mostly done; still need to add indexes for non-core projects, and
probably a bunch of content in the docs needs to be updated to refer to new
post-split module locations.

[itamarst]

t.i.utils has some SRV code that should be in twisted.names.

[radix]

Tv has added all of the indexes for the sub-projects. Hooray!!

[radix]

moving to urgent

[radix]

Hmm, I might as well move 2.0.txt to here.
* Extension module configuration for sub-packages.
  The idea is to have EXTENSIONS.cfg in each subpackage; it would have
  python code that constructs a list of Extensions in 'extensions',
  perhaps conditionally. This is not working because it seems that
  it'd require deep changes to zpkgtools. Need to talk to fdrake about this.
* docs. I'm thinking we should move them into twisted/*/doc, as we
  seem to be going that way for other stuff, and it would make
  packaging in zpkg easier. bin/ too.
* unit tests for zpkg usage...? I want to make sure stuff like
  extensions get build, and docs get included, but exarkun hates
  non-specific tests.
* finish refactoring/rewriting release-twisted to support releasing of
  all of our packages
* Poke Martin to release PyOpenSSL when 2.0 is almost out.
* Make sure there are no more urgent issues when 2.0 is released.

[jknight]

It seems sad to me to move all the non-code stuff into twisted/ because it makes it harder to find, and
makes it harder to do a manual copy install without getting extra crud.

[radix]

here's a draft requirements doc for 2.0 that I wrote for MFen (who seems to have
disappeared). It could be of general interest.

[radix]

storing files in buildbot sucks; I've put that doc (updated) into SVN as
admin/2.0.txt.

[radix]

we've stopped using zpkg. things are proceeding with distutils + helpers in
twisted.python.dist.

[glyph]

If I might ask, what preceded the move away from zpkg?
I am happy with the change, btw - I was hoping that distutils could be souped up
to provide the required mojo.  I just would like a more detailed explanation of
what's going on.

[radix]

zpkg had too much baggage for too little payoff -- it generated tarballs
straight from the go, so we couldn't customize it much.  Building extension
modules was very problematic, for multiple reasons: first, it didn't have a
flexible way to specify them, and second, because the layout of the tarballs
differs too much in strange ways from the source, so it was hard to actually
locate the .c files. (twisted conch was distributed with "twisted.conch" in the
tarball instead of "twisted/conch", for example).
We realized that zpkg was offering very little to us: automatic location of data
files and packages (so you don't have to remember to update your setup.py when
you add a non-.py file) and dependency-inclusion. (I thought it also did some
magic to install subprojects into existing package hierarchies, but it doesn't
-- it  works fine using vanilla distutils).
The first two (automatic location of data and packages) I've already implemented
in t.p.dist. Dependency inclusion won't be hard.

[radix]

What should the version numbers of each subproject be? I doubt they should
be 2.0, as none of them are at that mature a stage. I'm thinking of
starting them all off at 0.1.0, excepting twisted core.  If anyone
maintains a subproject and wants a different version, well, edit your
topfiles/setup.py file.

[itamarst]

Based on James' suggestion - as soon as you can make tarballs, even though w e
don't have debs etc., release 2.0a1 + a1 for all subpackages. Possibly prefixed
by "I am releasing 2.0a1 and 0.1a1 of all subprojects in two days, here's where
you can edit the version, metadata, etc. for your projects, and here's how you
add your project to the website" sent to mailing list.
This should help get the process going regarding debs, win32, etc..

[radix]

Ok. I've gotten to the point where I can generate tarballs for twisted core
and all subprojects. I still need to generate a sumo ball that includes
twisted, subprojects, and zope.interface. I'd also like to have a ball of
just Twisted Core + zope.interface.
I'll probably go with epydoc since spiv seems to be making sure it works.
I'll have to hook that up for each subproject, and the rest of the
automated stuff in release-twisted to deal with subproject balls. shouldn't
be too much work; if I can manage to get another three days of good
productivity, I can probably have a real release.

[Tv]

The prerelease tarballs, atleast core, seem to be missing LICENSE.

[Tv]

Please provide .tar.gz's, Debian is still stuck with gzip.
Please make the version number be less than 2.0 for the prereleases, something
like 1.99 or 1.99+2.0.0a1 is fine. dpkg --compare-versions is your friend.

[Tv]

The fact that e.g. TwistedConch's setup.py does "from twisted.python import
dist" means all the non-core twisted things build-depend(-indep) on twisted-core.
Which is no catastrophy, but people need to realize that to build twisted-foo,
they need to build _and install_ twisted-core. And that for now, I need to
always build the debs in a chroot :(

[Tv]

Please generate new prerelease tarballs, there have been a lot of bugfixes since
the tarballs were generated.
Is the way to generate them documented somewhere?

[Tv]

Words depends on Xish, and Xish depends on Words. Please break the cycle somehow.

[ralphm]

Xish should not have to depend on Words. The class _juserhost_Function defined
in xish does not really belong there as in abstract xml parse routines don't
have anything to do with Jabber IDs. It can be safely removed, thereby breaking
the cycle.

[ralphm]

_juserhost_Function was removed from xish/xpath.py by Tv in SVN. Dependency of
Xish on Words is thereby removed.

[Tv]

None of the tarballs in 2.0a3 seem to contain Twisted.Quotes. As there used to
be a twisted-quotes.deb, I was aiming to provide such a package in the future, too.

[Tv]

2.0a3 doesn't work with python2.2
cd . && /usr/bin/python2.2 setup.py install --root=debian/python2.2-twisted-core
--no-compile -O0
running install
running build
running build_py
error: build_py: supplying both 'packages' and 'py_modules' options is not allowed
make: *** [install/python2.2-twisted-core] Error 1

[Tv]

msg3511 is covered by #933
msg3502 is fixed by r13220

[radix]

Good bye!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!