cross-platform API for enumerating X509 certificates trusted by the platform for transport layer security

[glyph]

One component of #5445 (as originally discussed on #4023) would be an API for extracting the native trust roots from the platform. This is actually at least 3 tasks: one for Windows, one for Mac OS X, and at least one for Linux and BSD derivatives (although the only mechanism I'm familiar with there is the ca-certificates package in Debian, so perhaps there are other mechanisms we'd need to use as well).

I think that there's a way to discover the '/etc/ssl/certs' path (the one ca-certificates installs) via some API in OpenSSL, and if there is, we should use it, so that it will work with an arbitrary distro rather than being hard-coded to where Debian decided to stick it.

On Windows - and this is purely from a quick glance at the reference documentation, so take it with a grain of salt - I believe the right way to do this is to use  CertOpenSystemStore with the string "CA", or possibly "ROOT", or maybe both, and then do  CertEnumCertificatesInStore or maybe just  PFXExportCertStoreEx to dump the certs into a format we can import into OpenSSL.

On OS X, and again, I haven't done this, I believe you just have to call  SSLCopyTrustedRoots to get the default trusted SSL CA certificates and then  SecCertificateCopyData on the retrieved roots to turn them into DER (which we can then load into any SSL implementation).

[exarkun]

What does "inspect" mean in this context?

[glyph]

Replying to exarkun:

What does "inspect" mean in this context?

I changed it to "enumerate" since by "inspect" I just meant "present in a format that could be imported into an arbitrary TLS implementation so that we can do this before we figure out how to abstract away OpenSSL".

[exarkun]

I think that there's a way to discover the '/etc/ssl/certs' path (the one ca-certificates installs) via some API in OpenSSL, and if there is, we should use it, so that it will work with an arbitrary distro rather than being hard-coded to where Debian decided to stick it.

The way to discover this is probably via the OPENSSLDIR macro exposed by <openssl/opensslconf.h>.

[glyph]

Replying to exarkun:

The way to discover this is probably via the OPENSSLDIR macro exposed by <openssl/opensslconf.h>.

Thanks for the pointer. Is this macro exposed by PyOpenSSL?

[exarkun]

Replying to glyph:

Replying to exarkun:

The way to discover this is probably via the OPENSSLDIR macro exposed by <openssl/opensslconf.h>.

Thanks for the pointer. Is this macro exposed by PyOpenSSL?

Not by any existing version (<=0.13), no.

[glyph]

Replying to exarkun:

Not by any existing version (<=0.13), no.

Do you believe that it would be worth opening a pyopenssl ticket to expose it?

[exarkun]

Do you believe that it would be worth opening a pyopenssl ticket to expose it?

It seems like a useful thing to expose, and a ticket would, the very least, provide a place to discuss why it's actually the wrong idea. So, sure.

[glyph]

Done:  https://bugs.launchpad.net/pyopenssl/+bug/1138667

[itamar]

Do we actually need to expose OPENSSLDIR if we can just use Context.set_default_verify_paths? I guess it'll let us check if it's actually set to something sane, rather than my current default of "always use it on Linux".

[itamar]

I'm going to move code out of #4888 branch into here, and then limit its scope to platforms that support Context.set_default_verify_paths; I'm going to assume any reasonable Linux distribution will support this. Separate tickets will be opened for OS X and Windows.

[itamarst]

(In [37516]) Branching to 'trusted-ca-linux-5446'

[itamar]

I opened #6371 and #6372 for Windows and OS X native support. #6334 is the ticket for bundling our own file with CAs certificates.

[itamar]

I'm going to skip FreeBSD, and do it in separate ticket too (same code as Linux, just need FreeBSD-detection).

[itamar]

 http://buildbot.twistedmatrix.com/boxes-supported?branch=/branches/trusted-ca-linux-5446 should be running.

I'm not writing docs until the full set of tickets is done, I think.

[glyph]

Replying to itamar:

I'm going to move code out of #4888 branch into here, and then limit its scope to platforms that support Context.set_default_verify_paths; I'm going to assume any reasonable Linux distribution will support this.

[citation needed].

It is not a good idea to make security-critical assumptions about "common" distribution configurations. There's basically no way to figure out if set_default_verify_paths worked.

Granted, the failure mode is just that everything will fail to verify, right? So I guess it'll just be broken, not insecure.

[itamar]

The included unit tests verify that certificates signed with a newly generated CA are not validated, so "broken rather than insecure" is what you'll get, yes.

Once OPENSSLDIR is exposed, we can also verify the file is there.

[exarkun]

1. in the tests:
   1. It is probably obvious, but the approach taken by test_caCertsPlatformRejectsRandomCA is not the ideal approach. Since pyOpenSSL doesn't expose a non-Connection oriented way to invoke the verification logic, I can see that it is hard to exercise this code in any other way. However... isn't this just a test for Context.set_default_verify_paths? In other words, isn't this a test for a pyOpenSSL API which already has unit tests in pyOpenSSL?
   2. A corollary (or something) to the above may be that test_caCertsPlatformLinux should be trying a little harder to verify that it is faking Context correctly. If we were confident we were calling set_default_verify_paths correctly, and pyOpenSSL is confident that method works, then we don't need the integration-style test. Right?
   3. Please re-use some existing keys in the unit tests. Key generation isn't fast and doesn't need to happen each time the tests run.
   4. You could use dumpPEM instead of dump(..., FILETYPE_PEM)
   5. You could run all of the platform variation tests all of the time by parameterizing the platform object.
   6. test_caCertsPlatformRejectsRandomCA introduces some use of fail* methods. If somehow this test method doesn't end up going away, these should be assert* instead. fail* methods are going away.
   7. And use open instead of file :/
   8. Making _contextFactory an attribute of CertificateOptions seems nicer than making it a new "private" parameter of getContext (since positional parameters can't actually be private).
   9. Code should be explicit with bytes literals vs unicode literals
2. in the implementation:
   1. Adding PLATFORM as a possible value of caCerts precludes the possibility of using the platform certificates plus any custom certificates. It also doesn't help the chances of PrivateCertificate.options being usable with this option (I realize it is not now usable with it, but according to some stated goals for the API, this would be the preferred way to get a CertificateOptions - whether that is believable or not perhaps bears discussion, though). Anyhow, it also makes the docs confusing, because this sentence reads poorly: C{list} of L{OpenSSL.crypto.X509}, or L{CASources} constants.. Can it be a list containing CASources - no, it can't, but I expect it will take some people some time to figure this out.
   2. _makeContext is missing a docstring
   3. Do we have precedent for putting ticket numbers in exception messages? In comments in the source I can understand, but showing this to end users who misconfigure their SSL server seems like it could be another point of confusion.
   4. I'll try to bikeshed on the name CASources only this one time: these aren't really sources of CAs ("certificate authorities"). Perhaps they're sources of certificate authority certificates. So "CACertificateSources" would be a bit better. Of course, they're not necessarily actually event certificates from widely recognized certificate authorities. You may choose to configure your system with your own custom set of certificates. What this option really specifies is something to do with trust roots, I think, in the form of certificates. So perhaps a name involving either "trust" or "root" would be nice. Good luck inventing something elegant and intuitive with that.
3. Some narrative docs need to be updated to reveal this information to people - presumably the (already somewhat low-level) ssl howto would be a good place. Also, pointing out the pitfalls of the current implementation seems like a must. This never works except on Linux (and fails by preventing you from setting up an SSL connection), and on Linux it works only if you're using a build of OpenSSL which was properly configured at build time. And figuring out which of these situations you're in on a particular machine may not be easy for some people. Let's be proactive and try to keep people from digging themselves into too deep a hole.

Thanks!