Ticket #3696: irc_conversation_docs.txt

File irc_conversation_docs.txt, 14.6 KB (added by Chris Wolfe, 5 years ago)

irc discussion about docs

109:41 <herrwolfe45> I'm working on ticket #3696 and am a bit stuck on what type of documentation I should write - I'm thinking I should write a how-to describing how one should install twisted and it's various optional dependencies. This how-to would be in the narrative docs for twisted core. Does this sound like a decent idea?
209:41 <glyph> herrwolfe45: That sounds fantastic.
309:42 <herrwolfe45> glyph: excellent, I'll start with that - thanks
409:45 <herrwolfe45> ergh its
509:48 <exarkun> When writing that, don't forget that `pip install Twisted` is the least preferred installation method.  When possible people should prefer an OS-supplied package.  (But it's certainly not always possible.)
609:49 <exarkun> (Or possibly tailoring different sections of the document to different audiences makes sense - developers working on Twisted have slightly different requirements from developers using Twisted have very different requirements from end-users using an application that depends on Twisted.)
709:50 <glyph> exarkun: Least preferred by whom?
809:51 <glyph> exarkun: I would actually say "pip install Twisted" is the _most_ preferred installation method at this point.  OS-provided packages have kinda dropped the ball at this point as far as keeping up with releases and providing isolated, repeatable installations.
909:52 <glyph> The major reasons to prefer OS-vendor-provided packages are integrations with things like GTK and pyObjC, which are definitely not the most popular use-case :-\
1009:52 <exarkun> It's most preferred if it's most preferred, I agree - but that's also sort of a useless statement. ;)
1109:52 <exarkun> If your OS vendor has packaged the version of Twisted you need, you should use that package.
1209:52 <exarkun> That's why I said "when possible".
1309:53 <glyph> exarkun: Why?
1409:53 <exarkun> Because it's going to get security updates, for example.
1509:53 <glyph> exarkun: We ship security updates via pip as well, though.
1609:54 <exarkun> Yes, but the upgrade experience for pip isn't as good.
1709:54 <exarkun> You want unattended-upgrades to just install the security update - if possible.
1809:54 <exarkun> You want as few installations on a system as possible.  You don't want to have to remember to pip upgrade all of your deployments.
1909:54 <glyph> exarkun: Except unattended-upgrades is insufficient, because you need something that is going to restart your Twisted services for you when the updates are available; and your OS isn't going to help with that anyway.
2009:55 <exarkun> And (not security related) it's not going to screw with other OS-provided packages that have a Twisted dependency.  Perhaps those apps break when you install the newer Twisted into your environment (so, yay, virtualenv or whatever, but `pip` doesn't /require/ virtualenv).
2109:55 <glyph> exarkun: also, like, Docker completely breaks unattended-upgrades anyway, which I am still trying to figure out how to deal with :-)
2209:55 <glyph> Aah.
2309:55 <exarkun> glyph: You're basically saying "OS packages aren't 100% perfect so use pip"
2409:55 <glyph> Right.
2509:55 <exarkun> ignoring the fact that pip is also not 100% perfect
2609:56 <glyph> exarkun: Virtualenv is the thing, really, not pip.
2709:56 <exarkun> There are advantages and disadvantages to both.
2809:57 <exarkun> Maybe you're even right that on balance virtualenv & pip gives you the best trade-offs - but only if you understand how to manage it.  I'd argue that someone who understands those things deeply isn't going to read our installation documentation anyway: they already know how to do this.
2909:57 <exarkun> For the folks most likely to need this documentation (again, *if possible*) their OS packages are going to provide a better experience.
3009:58 <exarkun> And I'm not saying don't provide docs about both approaches, I'm just saying don't provide docs that unconditionally say "install Twisted with pip no matter what".
3109:58 <glyph> exarkun: OK, let's talk about the folks most likely to need this documentation.
3209:58 <glyph> My understanding of the target audience is that this is people with a cursory understanding of Python but no real understanding of packaging or the tradeoffs involved.
3309:59 <exarkun> I gave three example target audiences above
3409:59 <exarkun> I think your description is pretty close to the third example and it's probably the group most in need of any kind of installation docs, yes.
3510:00 <glyph> exarkun: My experience of that audience is that they know what "pip" is because they have 2 other dependencies they've already installed, both of which had installation documentation that told them to do "sudo pip install".
3610:00 <exarkun> (But the other groups are interested in the particulars of how they might get all of Twisted's dependencies since Python's packaging ecosystem is still immature that you can't *totally* rely on that going smoothly)
3710:01 <glyph> exarkun: Frequently they've also installed 2 or 3 versions of Python by downloading source code or installers from python.org.
3810:01 <exarkun> You're narrowing in on the *really* screwed end of the spectrum.
3910:02 <exarkun> Our docs can't be a comprehensive course in how not to screw up your computer, either.
4010:02  → iffy (was magmatt) joined
4110:02 <glyph> exarkun: Yes, and I don't want them to be.
4210:03 <glyph> exarkun: But, if we say "use your operating-system provided packages", this presupposes a level of knowledge of their operating environment that I think literally zero of the people in this audience (people who need some kind of set-up/installation documentation) actually have.
4310:03 <exarkun> Again, I don't think that's *all* we should say.
4410:04 <glyph> exarkun: I'm not arguing that we should just say "use pip, hth hand" either.
4510:05 <glyph> In fact I'm not sure that I'm arguing *for* anything in particular.  I think our preference for saying "use your OS installed packages" produces undesirable outcomes and I would like to explore what we should be saying instead.
4610:06 <exarkun> Okay
4710:06 <glyph> Maybe we should have an entire document that says "using the Twisted packages from your operating system"
4810:07 <glyph> The chief undesirable outcome I have noticed this guidance producing is people having really no idea what "operating system packages" are, and having to deal with multiple environments (almost always: OS X for dev, Linux for prod)
4910:07 <glyph> so they build some knowledge in one environment that completely fails to transfer to the other
5010:08 ⇐ itamarst quit (~itamarst@pool-108-20-253-103.bstnma.east.verizon.net) Quit: Leaving.
5110:08 <exarkun> That seems like the kind of basic misunderstanding that it might be very difficult for us to correct.
5210:08 <glyph> exarkun: well, this is the advantage of suggesting a virtualenv-based approach to installations.
5310:08 <exarkun> For one thing, someone who thinks they know about installing software because they've done it on OS X might not even think to look at our documentation to learn how to install things on Linux.
5410:09 <exarkun> glyph: Calling it an advantage seems a little tricky to me.
5510:09 <exarkun> It's definitely a trait.
5610:10 <exarkun> Offering one lowest-common-denominator solution is good because once someone learns it they don't have to learn something else but it's bad because maybe in learning something else they could produce a better result.
5710:12 <glyph> I'm not really sold on the OS-pavkages route being "a better result"
5810:13 <glyph> Thus far, the advantages we've discussed are: half of an automated security updates solution if you're using Debian, and reduced build interdependencies for desktop / GUI integration stuff
5910:15 <glyph> The advantages of virtualenv+pip are: portability, a greater degree of environment isolation and therefore predictability, and more knowledge sharing with other projects (since most other python projects recommend that deployment strategy)
6010:15 ⇐ khorn quit (~funsize@pool-72-64-101-86.dllstx.fios.verizon.net) Read error: Connection reset by peer
6110:16 <exarkun> Another factor worth mentioning, probably, is chances of actually producing a working installation
6210:17 <exarkun> virtualenv & pip have extra dependencies - eg, a C compiler and various headers
6310:17 <exarkun> If you actually want `pip install Twisted` to work in a virtualenv, better install libssl-dev (or is it libssl-devel) and other such things first
6410:18 <exarkun> (using your OS's package manager!)
6510:19 <glyph> Good point, the portability argument is weakened considerably by the totally non-portable nature of development dependencies
6610:20 <Alex_Gaynor> Isn't it a goal that twisted be installable without a C compiler?
6710:20 <glyph> Not just libssl-dev, of course: don't forget libffi-dev ;)
6810:20 <exarkun> Alex_Gaynor: Yes - but that's just Twisted. If you're installing Twisted and all of its dependencies, which you are if you use pip in a virtualenv, life is a lot harder.
6910:20 <glyph> Alex_Gaynor: Sure, but then you won't get SSL support (unless somehow cryptography is going to be installable without a C compiler too)?
7010:21 <exarkun> Right, so even in the best possible case, you end up with a crippled Twisted install - which may not be what you wanted.
7110:21 → khorn joined (~funsize@pool-72-64-101-86.dllstx.fios.verizon.net)
7210:24 <exarkun> So, getting practical, would our virtualenv & pip installation documentation include a section on how to use various OS's native package managers to install those dependencies?
7310:25 <glyph> exarkun: I think so.
7410:25 <hynek> that’s three lines to cover Linux & FreeBSE. OS X is the real issue because it’s a) unstandardized and b) used by a lot of junior peole
7510:26 <glyph> hynek: "type cc in a terminal and follow the on-screen instructions"
7610:26 <glyph> hynek: we _don't_ need to help you install PyGame and GTK development libraries, remember: libffi and openssl are both bundled with the platform
7710:27 <exarkun> hynek: But once you've got those three lines, you've brought into the extra conceptual overhead - and putting a "python-twisted" package name somewhere on those lines doesn't increase that burden (and you get to stop there instead of talking about virtualenv & pip if you want)
7810:27 <exarkun> So *some* of the portability advantage has been lost
7910:27 <glyph> exarkun: except you _might_ not get to stop there, because your *other* dependencies, which you want to integrate with Twisted, may not install cleanly in that environment (or may require the use of 'sudo pip')
8010:28 <exarkun> glyph: Sure, but some people will get to stop there.
8110:28 <exarkun> This suggests to me the docs should be something like
8210:29 <hynek> once people start using python-twisted, they start installing crap into their site-packages and then they come crying
8310:29 <glyph> hynek: let's be specific :-).  What does "come crying" mean?  What breaks?
8410:30 <hynek> glyph: https://hynek.me/articles/virtualenv-lives/ the examples are extra-twistedish but the points remain
8510:30 <exarkun> "Maybe this OS-specific command is sufficient for you: <dpkg / yum / whatever to install Twisted>.  If you need a newer version, start with this OS-specific command: <dpkg / yum / whatever to install Twisted's dependencies> and then <virtualenv & pip instructions instructions>"
8610:30 → itamarst joined (~itamarst@pool-108-20-253-103.bstnma.east.verizon.net)
8710:30 <exarkun> hynek: People will /have/ to install some native packages.
8810:30 <hynek> it’s not just about newer versions. it’s also about klein, treq etc pp
8910:30 ⇐ trenton42 quit (~trenton@ Ping timeout: 240 seconds
9010:31 <hynek> exarkun: that statement has absolutely nothing to do with my point. not sure whether it’s my fault or yours though. :)
9110:31 <exarkun> hynek: Given that, I don't think you can get away from your concern about people incorrectly installing unpackaged software in system paths.
9210:32 <glyph> exarkun: I think there's an unspoken assumption here: the number of native packages required from the platform is small (literally 2: openssl-dev and libffi-dev); the number of python packages required by application developers is large (klein treq characteristic txamqp txkazoo ampoule service_identity pyasn1_modules and so on and so on)
9310:33 <exarkun> Or maybe not "If you need a newer version" but "If you'd prefer to have an isolated installation (for example, so you can have multiple versions of Twisted) ..."
9410:33 <exarkun> glyph: It's larger than that.
9510:33 <exarkun> glyph: And even if it's just one thing, it's the conceptual overhead of switching to another tool.
9610:33 <glyph> exarkun: Perhaps I should have said "typically"; for some developers it's huge. What am I forgetting, though?
9710:33 <exarkun> The number of arguments you copy/paste along with that tool's invocation command is sort of irrelevant.
98New messages since you tabbed out
9910:34 <glyph> exarkun: On OS X and Windows, you don't actually need to switch to another tool though
10010:34 <glyph> exarkun: on Windows, cryptography wheels address the need for openssl and cffi wheels address the need for libffi
10110:34 ⇐ itamarst quit (~itamarst@pool-108-20-253-103.bstnma.east.verizon.net) Client Quit
10210:34 <dstufft> glyph: also python-dev
10310:34 <glyph> exarkun: on OS X, "install the developer tools" is a single operation you do once and almost _certainly_ need to do to get a lot of things done
10410:35 <exarkun> Are there binary wheels for cryptography now?
10510:35 <hynek> I think jp’s point is basically that there should be as little hurdles as possible to try Twisted which I completely agree with.  Although if documented, it should be pointed out that it’s just a sneak-peak and if pursuited for anything else than writing distribution-specific software, it will end in tears.
10610:35 <glyph> exarkun: for windows, yes!
10710:35 <glyph> exarkun: it is ~~~amazing~~~
10810:36 <exarkun> Okay, cool.  I didn't notice when that happened.
10910:36 <glyph> exarkun: they statically link openssl, you don't need to go to that shitty website any more
11010:36 <hynek> it’s still impossible to build linux binary wheels :(
11110:36 <exarkun> Right.  But you do need to pip upgrade every time an OpenSSL security update comes out (~ once a month)
11210:36 <glyph> exarkun: Yup.
11310:36 <exarkun> But it's Windows.  So too bad, that's what you have to do.  We don't have to argue about that.
11410:37 <glyph> exarkun: Or it's Docker and that's what you have to do too ;-) (seriously somewhere else we should have a conversation about container security updates because I do not know what the _heck_ is going on with that)
11510:37 → itamarst joined (~itamarst@pool-108-20-253-103.bstnma.east.verizon.net)
11610:38 <exarkun> Okay I gotta do some work.
11710:38 <hynek> glyph: there is none. my tour into docker made me drink.
11810:38 <glyph> exarkun: I probably should too.
11910:38 — glyph wonders if he can capture some notes from this conversation in a useful way