[Twisted-web] Making Secure HTTPS requests, SSL Method, and Certificate Management using Twisted

[Glyph]

On Oct 6, 2014, at 7:55 AM, Carl Waldbieser <cwaldbieser at gmail.com> wrote:

> I have a couple projects I am working on where I would like to make HTTPS requests using Twisted.  I reviewed the articles "Using TLS in Twisted" [1] and "Using the Twisted Web Client" (section "HTTP over SSL") [2].  It seems like various options exist that will allow me to make HTTPS requests using the CA certs bundled on the client OS.

Thank you for diligently studying all of our existing documentation on the subject.  I apologize for its deficiencies.  This API has grown in fits and starts over the last decade or so and it has not ended in the most consistent shape.  Hopefully we can improve this based on your feedback for the next release.

> I would like to be able to tell my HTTPS clients to accept specific *internal* CA certificates *in addition* to the certificates provided by the OS.  Initially, I thought this might be possible by passing a custom t.w.c.BrowserLikePolicyForHTTPS to the t.w.c.Agent as its `contextFactory` argument. I wasn't quite sure how to go about doing this, so I got some advice on StackOverflow [3]. With some slight modifications to the solution presented there, I was able to create a custom Trust Root that accepted a list of paths to CA cert files in PEM format that I wanted to add to the client. The custom trust root is passed to the BrowserLikePolicyForHTTPS. The policy is passed to the Agent.
> 
> The one hitch is that the IOpenSSLTrustRoot interface upon which my custom trust root is based is located in `twisted.internet._sslverify`[4], which if I understand correctly, is a private module and not supposed to be used as an API.  Is there a *supported* way to specify *additional* CA certs to use during SSL verification when making HTTPS requests using Twisted? If so, what is the recommended method?

Well, the "good" news is, on OS X, if you're using the bundled system OpenSSL, you can't turn this behavior off: <https://hynek.me/articles/apple-openssl-verification-surprises/>.  Although we may eventually ship a mitigation for this vulnerability in Twisted, so probably best not to rely on that long-term :).

The bad news is that, no, we really had two use-cases in mind here; either:

your software is trusting the trust cartel as specified by the operating system vendor and/or the user, and your software is opting out of any trust configuration, or
your software has a specific understanding of its trust root and is specifying it explicitly, because it knows who it expects the peer to be signed by

So while your use-case sort of makes sense to me, it didn't come up in the last round of SSL enhancements and there's no straightforward way to go about this.

However, it should nevertheless be possible to do it without delving into the private API.

If you write your own IOpenSSLClientConnectionCreator, like this:

from twisted.internet.interfaces import IOpenSSLClientConnectionCreator
from twisted.python.components import proxyForInterface

class AddExtraTrustRoots(proxyForInterface(IOpenSSLClientConnectionCreator)):
    def __init__(self, extraTrustRoots, original):
        self._extraTrustRoots = extraTrustRoots
        super(AddExtraTrustRoots, self).__init__(original)


    def clientConnectionForTLS(self, tlsProtocol):
        connection = (super(AddExtraTrustRoots, self)
                      .clientConnectionForTLS(tlsProtocol))
        cert_store = connection.get_context().get_cert_store()
        for cert in self._extraTrustRoots:
            cert_store.add_cert(cert)
        return connection

you can delegate to the pyOpenSSL API for manipulating the trust settings of the connection, although Twisted will not provide a high-level wrapper for this.

You can then use the above wrapper like:

options = AddExtraTrustRoots([OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_PEM, "..."), ...], optionsForClientTLS(u"google.com"))

I haven't tested this, let me know if I've overlooked something terrible :).

> Another related concept that was not clear to me is how one might specify the SSL method (e.g. SSLv23_METHOD, SSLv3_METHOD, etc.) when making the request. Is there some recommended way to pass options to indicate the SSL method that ought to be used?

The default configuration allows for TLSv1.0, TLSv1.1, and TLSv1.2 (insofar as those versions are supported by your OpenSSL).  Do you really, really, really need to support worse protocol versions than that?

You don't actually want to change the "method", as "method" is a bizarre fiction of the OpenSSL API and is not actually how the protocol works; SSLv23_METHOD is the only "method" that allows protocol version negotiation, so you always want to start there and then set/unset the appropriate OP_NO_* options for the protocol versions you want to exclude.

This is all pretty tedious and gross so if you're sure you want to do it I will describe how in the next reply :).

> Any guidance would be appreciated.

I hope that this was sufficient!

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-web/attachments/20141007/b5f60367/attachment.html>