Opened 3 years ago

Last modified 9 months ago

#6371 enhancement new

Support native Windows trusted CA database for SSL certificate validation

Reported by: itamar Owned by: aronbierbaum
Priority: normal Milestone:
Component: core Keywords:
Cc: adiroiban Branch: branches/windows-trust-root-6371
branch-diff, diff-cov, branch-cov, buildbot
Author: glyph


This was originally part of #5446, where Glyph wrote:

On Windows - and this is purely from a quick glance at the reference documentation, so take it with a grain of salt - I believe the right way to do this is to use CertOpenSystemStore with the string "CA", or possibly "ROOT", or maybe both, and then do CertEnumCertificatesInStore or maybe just PFXExportCertStoreEx to dump the certs into a format we can import into OpenSSL.

Attachments (3)

windows_trust_root.patch (4.4 KB) - added by aronbierbaum 21 months ago.
windows_trust_root2.patch (4.4 KB) - added by aronbierbaum 21 months ago.
windows-trust-root-6371-2.patch (4.4 KB) - added by aronbierbaum 21 months ago.

Download all attachments as: .zip

Change History (17)

comment:1 Changed 3 years ago by exarkun

There is a Python library to help out with this too:

This may be easier than writing C, or Cython, or using ctypes or cffi. Or maybe not, I haven't investigated much. But it's something to look at I guess.

comment:2 Changed 2 years ago by itamar

wincertstore is written with ctypes.

comment:3 Changed 2 years ago by aronbierbaum

I have been working on an implementation that uses for ctypes also. In the spirit of not duplicating effort, is anyone else also working on a solution?

comment:4 Changed 2 years ago by itamar

Assuming wincertstore does what it says it does, presumably that's who you should talk to. Unless you want to add something to Twisted, in which case using wincertstore seems like a good start.

Changed 21 months ago by aronbierbaum

comment:5 Changed 21 months ago by aronbierbaum

  • Keywords review ssl added

I have added an initial pass at adding support for certificate verification on Windows using the wincertstore package. Please provide feedback and required changes.

Changed 21 months ago by aronbierbaum

Changed 21 months ago by aronbierbaum

comment:6 Changed 19 months ago by glyph

  • Keywords ssl removed

We use keywords for workflow, and ssl is not a workflow state. Therefore, removing that keyword.

comment:7 Changed 19 months ago by glyph

  • Author set to glyph
  • Branch set to branches/windows-trust-root-6371

(In [43870]) Branching to windows-trust-root-6371.

comment:8 Changed 19 months ago by glyph

  • Keywords review removed
  • Owner set to aronbierbaum

Hi aronbierbaum! Thanks very much for this contribution - it's an important building block of Twisted's client TLS subsystem, and I'm glad someone is working on this :).

Unfortunately this first cut at the patch looks like it's not in great shape:

  1. there are a number of coding-standard violations.
  2. some of the tests are failing on some platforms
  3. wincertstore should really be included in the list of setuptools extras for the tls extra on Windows.

However, the general idea of using wincertstore is pretty good, and it looks like the implementation is basically correct. I'm looking forward to re-reviewing this!

comment:9 Changed 17 months ago by adiroiban

  • Cc adiroiban added

comment:10 Changed 14 months ago by glyph

A bug report in treq indicates that we really need to come up with a solution for this or people can't really use TLS on Windows; setting up a certificate bundle is incredibly subtle and undocumented.

comment:11 Changed 14 months ago by glyph

The specific issue of "empty trust store" as opposed to windows API bindings for real trust roots filed here.

comment:12 follow-up: Changed 13 months ago by akgood

Two notes:

  1. As of Python 2.7.9 (along with recent versions of 3.x), wincertstore is not needed; that functionality is available as "ssl.enum_certificates()" in the standard library.
  1. However, both wincertstore and the Python stdlib implementation have a significant caveat:

Starting with Vista, Microsoft began shipping only a very minimal set of root CA certificates with Windows. Microsoft does trust many other authorities, but for these, Windows relies on the "Update Root Certificates" feature:

"... if the application is presented with a certificate issued by a certification authority in a PKI that is not directly trusted, the Update Root Certificates feature (if it is not turned off) will contact the Windows Update Web site to see if Microsoft has added the certificate of the root CA to its list of trusted root certificates. If the CA has been added to the Microsoft list of trusted authorities, its certificate will automatically be added to the set of trusted root certificates on the user's computer."

Critically, this update mechanism is only invoked if you're using (e.g.) CryptoAPI functions to validate a specific chain; if you just ask Windows to enumerate the certificates it knows about, it won't pull anything down from Windows Update.

(Some concrete numbers: on a clean installation of Windows 8.1, running certmgr.msc shows 18 certificates listed in the "Trusted Root Certification Authorities"; by contrast, OS X comes with over 200 trusted roots).

As far as I understand it, the only 100%-accurate way to validate certificates against Windows' trust policy would be to hook the OpenSSL verify callbacks to retrieve the leaf and intermediate certificates provided by the server, then use CryptoAPI functions (e.g. perhaps CertGetCertificateChain?) to have Windows perform the actual chain validation.

(I filed a bug report about this in Python as

Last edited 13 months ago by akgood (previous) (diff)

comment:13 in reply to: ↑ 12 Changed 13 months ago by glyph

Replying to akgood:

Two notes:

Thank you very much for these highly informative implementation notes, akgood.

ideally, eventually, we can just use the CryptoAPI/NG functions for doing TLS directly, but until we do, hooking up the verify callbacks in the manner described seems like the best option. Except... presumably that will block (and the verify callback has to block, I think?)

comment:14 Changed 9 months ago by johnrocs

Some additional information regarding wincertstore.

My reading of the runes relating to automatic and dynamic updates is as follows:

In an organisation with a sysadmin, dynamic updates will be turned off and an internal mechanism will be in place to update clients.

Dynamic updates represents too much of a security risk.

There is also an option from Microsoft on a daily basis to automatically update the certificate store

So sysadmins can either arrange to have the daily update occur or have it downloaded to a central system under their control where they distribute the update plus any other certificates they need to add for company use.

In smaller organisations no doubt the automatic update will be left in place.

It would seem that the dynamic update is part of windows os and the application is not involved in the process. The verification either succeeds or fails.

As an example see this link that contains this info:--

<<Today Secunia PSI refused to run with the message: "an error occurred while verifying the security certificate". Then I found that IE refused o show because the certificate was "not issued by a trusted certificate authority".

Firefox did not have a problem with that webpage.

For some reason, IE did not recognize the "Thawte Server CA" certificate. IE also refused to recognize StartSSL.

This was really weird, because as far as I know, Windows 7 is supposed to automatically update root certificates.

Microsoft even explains how the process works in Vista. My first thought was that my firewall was blocking the update, but that was not it.

Event log showed event 4100 from CAPI2, which is "Successful auto update retrieval of third-party root certificate".

The problem was event 4110: "Failed to add certificate to Third-Party Root Certification Authorities store with error:

A certificate chain could not be built to a trusted root authority."

I manually downloaded and installed the latest root certificate update from Windows Update.

After that, everything works. I'm just left wondering why I had to deal with this in the first place.

A later comment says:

I think this may be because Cryptographic Services (CryptSvc) was unable to access the Internet because of the firewall.

Its description says that it includes the "Automatic Root Certificate Update Service, which retrieves root certificates from Windows Update".

There are CAPI2 events relating to downloading and unpacking a root certificates .CAB file, and those do not appear in my event log.>>

So windows cryptographic services does the dynamic check for certificate verification.

However, you can always download the latest certificate store and update it yourself.

This strikes me as similar to using the firefox download as happens on many linux systems and how the requests library works.

I did a simple test to see if the dynamic update works on my win7 system.

I used certmgr.msc to delete the GoDaddy root certificate.

I then opened a browser to the godaddy site to see what would happen.

In some side bar I saw a link that said the cert was failing but the main site site connected without problems.

When I checked the certificate store there was still no godaddy root certificate. So what was happening?

I then downloaded the update and updated my store. I ended up with about 200 entries. More than before but since I did not make a note I can't really be sure if I got more than when I started.

I just need to run some more tests to see how this dynamic update is supposed to work.

Manually updating the certificate store

The link to manually update the certifcate store is below:

To Manually install the certificates

  1. Download
  1. Extract the files using the command rootsupd.exe /c /t:C:\temp\extroot
  1. from c:\temp\extroot run the following 4 commands (from an elevated prompt)

updroots.exe authroots.sst

updroots.exe updroots.sst

updroots.exe -l roots.sst

updroots.exe -d delroots.sst

Here is an extract from microsoft technet explaining how the automatic update works.

<<The Update Root Certificates feature in Windows Vista is designed to automatically check the list of trusted authorities on the Windows Update Web site when this check is needed by a user's application. Specifically, if the application is presented with a certificate issued by a certification authority in a PKI that is not directly trusted, the Update Root Certificates feature (if it is not turned off) will contact the Windows Update Web site to see if Microsoft has added the certificate of the root CA to its list of trusted root certificates. If the CA has been added to the Microsoft list of trusted authorities, its certificate will automatically be added to the set of trusted root certificates on the user's computer. The Update Root Certificates feature can be turned off in Windows Vista by using Group Policy.

For more information, see "Procedures for Viewing or Changing Group Policy Settings that Affect Certificates in Windows Vista," later in this section.>>

Below is a link to a mozilla forum where the sysadmins have been complaining over the past 7 years that firefox will not support wincertstore so it can be integrated into their control systems.

As a result they drop firefox and only support chrome and IE.

Note: See TracTickets for help on using tickets.