[Twisted-Python] Python3 twistd daemon for Ubuntu 14.04 alternatives

[Glyph Lefkowitz]

> On Feb 22, 2017, at 10:16 AM, Phil Mayers <p.mayers at imperial.ac.uk> wrote:
> 
> On 22/02/17 17:42, Hynek Schlawack wrote:
> 
>> I have to disagree here:  I don’t want build tools of any kind in my
>> final containers therefore I build my artifacts separately no matter
>> what language.  Of course you can just build the venv on your build
> 
> Agreed, 100%. Apologies if I gave you the impression I was advocating otherwise.
> 
>> server without wheeling up a temporary container and then package it
>> using Docker or DEB or whatever.  You should be separating building
>> and running anyway so Python – as much as I’d like Go-style single
>> binaries too – is in no way special here.  The nice thing about
>> temporary containers though is that I can do all of that on my Mac.
> 
> I agree that you need to separate building and installation, and I've got no particular beef with using a container, chroot, throwaway VM or whatever works for people in doing the build phase.
> 
> (What people do with the resultant build output - and in particular whether there is a lot of ignoring of the hard-learned lessons of system package managers going on now - I will not comment on ;o)
> 
> What I was trying to say - badly, apparently - was that the system python *could* be attractive to someone because many dependencies may exist in the OS package list in suitable form, but conversely may not exist in PyPI in binary form for Linux.

Yes, and building these binary artifacts is often harder than some people (cough, alpine, cough) seem to think.  But there are better ways to square this circle than restricting yourself to the versions of python libraries that happen to be available in your distro.

> As a very simple example: if you have a traditional (non-container) Linux system hosting a Python application in a virtualenv, and you deploy a Python app to a virtualenv e.g. using Puppet or Ansible, you either need to:
> 
> 1. Use no C extensions
> 2. Hope there's a manylinux1 binary wheel
> 3. Use the OS package and --system-site-packages
> 4. Compile the C extensions and make them available to pip
> 
> #2 seems useful now that I know about it but - correct me if I'm wrong - the manylinux1 permitted C dependencies are super-tiny, and would not permit e.g. cryptography or psycopg2?

Cory already pointed this out tangentially, but I should emphasize: 'cryptography' and 'psycopg2' are things that you depend on at the Python level.  The things you depend on at the C level are libssl, libcrypto, and libpq.  If you want to build a manylinux wheel, you need to take this into account and statically link those C dependencies, which some projects are beginning to do.  (Cryptography _could_ do this today, they already have the infrastructure for doing it on macOS and Windows, the reason they're not shipping manylinux1 wheels right now has to do with the political implications of auto-shipping a second copy of openssl to Linux distros that expect to manage security upgrades centrally).

> #4 is what you are advocating for I believe? But can we agree that for smaller projects, that might seem like a lot of repeated work if the package is already available in the OS 

If you're going to do #4 with dh_virtualenv, your .deb can depend on the relevant packages that contain the relevant C libraries, and build linux wheels, which are vendor-specific and can dynamically link to whatever you want (i.e. not manylinux wheels, which are vendor-neutral and must statically link everything).  Manylinux wheels are required for uploading to PyPI, where you don't know who may be downloading - on your own infrastructure, where you are shipping inside an artifact (like a .deb) that specifically has metadata describing its dependencies, "linux" wheels are fine.  Alone, hanging around on PyPI as .whl files rather than as .debs in your infrastructure, they'd be mystery meat, but that is not the case if they have proper dependency metadata.

It might seem weird to use Python-specific tooling and per-application vendoring for Python dependencies, and yet use distro-global dynamic linking for C dependencies.  But, this is actually a perfectly cromulent strategy, and I think this bears a more in-depth explanation.

C, and particularly the ecosystem of weird dynamic linker ceremony around C, has an extremely robust side-by-side installation ecosystem, which distros leverage to great effect.  For example, on the Ubuntu machine sitting next to me as I write this, I have libasan0 (4.8.5) libasan1 (4.9.4) libasan2 (5.4.1) *and* libasan3 (6.2.0) installed, and this isn't even a computer with a particularly significant amount of stuff going on!  Nothing ever breaks and loads the wrong libasan.N.

Python, by contrast, tried to do this in a C-ish way, but that attempt resulted in this mess: https://packaging.python.org/multi_version_install/ <https://packaging.python.org/multi_version_install/>, which almost nobody uses.  Right at the top of that document, "For many use cases, virtual environments address this need without the complication ...".

Even if you are 100%, completely bought into a distro-style way of life, no containers at all, everything has to be in a system package to get installed, virtualenvs still make more sense than trying to sync up the whole system's Python library versions.

The reason nobody ever went back and tried to do multi-version installs "right" with Python is that the Python and C library ecosystems are fundamentally different in a bunch of important ways.  For one thing, Python libraries have no such thing as an enforceable ABI, so coupling between libraries and applications is much closer than in C.  For another, no SOVERSION.  Also, many small C utilities (the ones that would be some of the smaller entries in requirements.txt in a Python app) are vendored in or statically linked in applications, so the "dependency management" happens prior to the container build, in the source repo of the upstream, where it is hidden.  Python dependencies often have a far higher rate of churn than C dependencies because of the ease of development, which means both more divergence between required versions for different applications, and more benefits to being up-to-date for the applications that do rev faster.

Finally, the build process for Python packages is much simpler, since they're usually treated as archives of files that move around, rather than elaborate pre-build steps that are often required for C libraries to make sure everything is smashed into the .so at build time.

So think of your Python libraries as "vendored in" to your package for these reasons, rather than depended upon in the OS, and then participate in the broader distro (i.e. "C") ecosystem by building wheels that dynamically link whatever distro-level dependencies they need to.

> Wondering out loud, I guess it would be possible for OS-compiled python extensions to be somehow virtualenv or relocation-compatible. One could envisage something like:
> 
> virtualenv t
> . t/bin/activate
> pip syspkg-install python-psycopg2
> 
> ...and this going off and grabbing the OS-provided dependency of that name, extracting it, and deploying it into the virtualenv, rather than the system Python.

This is sort of what dh_virtualenv is.  It doesn't set up the mapping for you automatically, but you can pretty quickly figure out that python-psycopg2 build-depends: libpq-dev.

> There are doubtless all sorts of reasons that is not practical.

The main thing is just that you have to decide on a standard format for distro-specific metadata, and then go encode it everywhere.  I'm pretty sure that distutils-sig would be open to codifying such an extension to the list of standardized metadata fields, so that tools can use it.

> Anyway, to be clear - I'm not advocating using the system Python. I'm trying to explain why, based on the efforts we expend locally, it could seem attractive to smaller sites.

To be even clearer, using the system python is fine - it's using the global python environment that has the most significant problem.

(Although, of course, the "system python" is probably CPython, and in most cases you want to be using PyPy, right?  So yeah don't use the system Python.)

I hope this explanation was helpful to those of you deploying with distro tooling!

-glyph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20170223/39b815b7/attachment-0002.html>