[Twisted-Python] Twisted Plugins - Implementation Discussion

[David]

On 04/22/2011 03:03 PM, Glyph Lefkowitz wrote:
>
> On Apr 21, 2011, at 9:01 PM, David wrote:
>
>> You can take a look at bento, which is my own response to the distutils
>> issues we have in the scipy community (but I would expect twisted and
>> most big python libraries to have similar issues):
>>
>> http://cournape.github.com/Bento/
>>
>> It is designed from the ground up with the idea of reliable
>> customization and complex build supports. It can already build numpy and
>> scipy with a near 50 % reduction in LOC compared to our setup.py, and
>> more reliably thanks to using a real build tool in the backend (waf, but
>> you can add support for a different one if you want).
>
> This looks /very/ interesting. You kind of bury the lead on that web
> page though, so let me excerpt it for those who saw the first line,
> thought "oh, it's yet another python packaging thing" and stopped
> reading in disgust:
>
>     Even better, bento has a distutils compatibiliy layer
>     <http://cournape.github.com/Bento/html/transition.html> so that you
>     can write a simple setup.py *which works under pip or easy_install*.
>
>
> (emphasis mine)
>
> This suggests that Twisted could actually /switch/ to Bento without
> creating a massive disruption for our users who want to install it with
> an existing automation tool - which, frankly, is the main use-case for
> distutils at this point.
>
> Would you recommend that we do this? Would there be a benefit? I like
> the part where you said "near 50% reduction in LOC" quite a lot but I'm
> sure there are issues that would come along with it.

Actually, twisted is on my list of packages to convert to bento to get a 
feeling of what's missing in bento :) To get an actual idea of what it 
is looking ATM, you can see here: 
https://github.com/cournape/numpy/tree/bento_waf_build (bento.info and 
bscript - especially numpy/core/bscript).

The reason why the distutils compatibility thing is not emphasized is 
because I cannot possibly support what makes bento interesting in my 
mind under this mode: out of tree builds, hooks support, recursive 
description, pluggable build backend, etc... After all, if I could 
support what I have in mind with distutils, I would have started from 
distutils and not from scratch (I already did in a former life, and 
bento is born out of that failure).

Now, concerning the use of pip/easy_install: if those tools's author 
were willing to add hook to support additional tool, it would not take 
much. Fundamentally, you only need to say "bentomaker install" instead 
of "python setup.py install", plus all the dirty details. This will be 
needed anyway with the distutils2 effort, since they have also started 
using a static format description and python setup.py install will not 
work anymore (IIRC).

As for what you would gain doing so:
	- robust recursive support (things like relativeTo as used in twisted 
are inherently fragile once you don't want to assume source tree == 
current directory)
	- automatic dependency handling, easy customization and parallel 
support for compiled code (waf automatically scan sources to find header 
dependencies - waf has its quircks, but it has been recently used by 
SAMBA, which is a pretty good endorsement in my mind as far as complex 
builds go)
	- a simple and robust way to install data files (install things in 
chroots will finally be possible, a pet-peeve of mine when deploying 
twisted apps)
	- it would help me evangelizing bento :)

As you mentioned, most people say "sigh, another packaging python 
thing". I have not found a good angle to quickly describe what I am 
doing, because it is more about the how than the what. Maybe you could 
say that bento is trying the "pylons approach" of reusing existing 
tools, whereas distutils is more of the "django approach". For the scipy 
community (where I am coming from), the advantages of using a real build 
tool with dependency handling is obvious, but many people don't care 
about that.

Note that bento is currently a moving target, but I hope to be close to 
a first alpha in a couple of months (bento development has started in 
december 2009). I am careful with timing because I don't want to 
reproduce the precedent of setuptools which became popular but with 
issues that became too costly to fix afterwards.

cheers,

David