[Twisted-Python] Deprecating subproject packaging

[Glyph]

> On Apr 22, 2015, at 09:28, Adi Roiban <adi at roiban.ro> wrote:
> 
>>>> So, I think we should:
>>>> 
>>>> - Deprecate the subproject setup.pys starting with 15.2 and stop packaging subprojects separately as of 16.0 (15.2 in May, 15.3 in July, 15.4 in September, 15.5 in November, 16.0 in January 2016 -- so ~4 releases + 1 major release notice).
>>>> - Remove the support code for subprojects in twisted.python._release + the release scripts in the 16.0 release.
> 
> I am ok with deprecating this ... but I hope that this will not
> discourage the effort to decouple the Twisted subprojects.

There is not really any such effort underway.  However, if anything, this change would make such an effort easier.  If we want to release the projects separately, we should maintain them separately; the lesson of this "subproject" split is that what users want, to perceive that projects are more "lightweight", is a different "front door" to each project - different websites, different documentation indexes, different bug trackers - and not necessarily just a smaller download.  I'm pretty sure that if we split everything up but the actual releases we'd have better luck convincing people twisted is not hard to understand.

One immediate thing I think we should decouple, though, are our native dependencies.  IOCPReactor should remain in Twisted, but iocpsupport should be a separate, optional dependency, available as an extra.  Similarly, _initgroups, sendmsg, portmap, and raiser should all be optional dependencies, so that Twisted itself can be a pure wheel, and you can opt in to enhanced functionality by installing extras.

> I would like to see the subproject developed as independent
> projects... this should help enforcing the decoupling and would speed
> up the test suite.

Speeding up the test suite should be accomplished by actually fixing and optimizing the tests, not by chopping big sections of it off.  The amount of tests we have right now can run very quickly; but there are two immediate problems:

1. we are not leveraging twistd -j, and since we have no buildbots that run it, we keep making changes that break it for our test suite.  We should introduce a builder that runs the tests in parallel.
2. many of the tests still return Deferreds when they should not, spin the reactor, wait for real timeouts, and so on.  we should endeavor to reduce the number of real-IO tests as much as possible.

> Right now if you make a change in twisted.positioning, buildbot will
> run all tests :(

Maybe we could speed up twisted.positioning's tests without sacrificing quality, but we would need to continue to run twisted.positioning's tests for every change to twisted core regardless, even if they were separate projects, or we would be reducing the amount of integration coverage received by twisted core.

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20150422/94056d21/attachment-0002.html>