Last week I helped to get the DigiNotar incident addressed in Mozilla's applications, and also in the NSS library. Today I was asked to explain in detail what we did inside NSS. Thanks a lot to Gervase Markham who helped as an editor of the following text.


This is an interim statement, and represents the personal understanding of me, Kai Engert. It has not been reviewed by other members of the NSS team. If necessary, a checked version of this message will be provided after the US/Canada public holiday. However, I have been involved in creating the patches that we used to address the DigiNotar incident.


Summary:
The NSS team has issued a new version of the NSS module that contains
trust information for CA certificates, NSSCKBI version 1.87. We believe it removes all trust in the DigiNotar root and in all known
cross-certificates and in the DigiNotar-controlled intermediates in the Staat der Nederlanden hierarchy. We have shipped a new release of NSS, containing the same code as the previous release and the updated trust store. It can be found here:

ftp://ftp.mozilla.org/pub/mozilla.org/security/nss/releases/NSS_3_12_11_WITH_CKBI_1_87_RTM/


Details:
I have been asked about the NSS crypto library. Because of this, I will skip what has been added at the application level in Mozilla's products (on top of NSS).

Note that the NSS library consists of several modules, which use their own sub-version numbers. For example, NSS contains a binary module that embeds CA certificates and trust information, according to the Mozilla CA policy. This list is stored in a binary module named NSSCKBI, and certificates within it are referred to as builtin certificates.

For the DigiNotar incident, we didn't produce a new NSS library release. Instead, we released newer versions of the NSSCKBI module.

Last week, in our immediate reaction, we simply removed the "DigiNotar Root CA" certificate and its trust from the NSSCKBI module. We used version number 1.85 for that older release of NSSCKBI. This release wasn't published separately, but it was used in Firefox version 6.0.1 and other Mozilla updates released at the same time.

Later, we noticed this approach wasn't sufficient, because other
intermediate CA certificates exist that are cross-signed by other
non-DigiNotar CAs that we still trust. We started to work on a better
approach, but this was delayed until Mozilla made final decisions
regarding the intermediates being used by the Dutch Government.

On Friday/Saturday, after the decision was made to completely remove all trust from CA certificates related to DigiNotar, the following approach was used as a better blocking mechanism.

We attempted to identify as many CA certificates as possible, and
inspected each of them carefully. Because the NSS library does not yet have the ability to actively mark a specific certificate as completely untrusted, in a way to prevent other trust paths to become active, we used a workaround.

We manually manipulated the binary structure of the known CA
certificates, and created special knockout certificates. The following procedure was used to create them:

- start with the original certificate

- manipulate the serial number to a new number, that is unlikely to
collide with other existing certificates (we used 0x*FFFFFFF)

- manipulate the NotBefore and NotAfter embedded in the certificates,
change them to be in the future when compared with the original ones

This has the following effect: When the NSS library attempts to verify a certificate, it will search the list of known/available certificates. If there are multiple candidates with the same subject names, NSS will prefer the ones that are more recent. This means, our knockout certs will be preferred.

Because of an implementation detail of NSS and Firefox, we made an
additional change to the certificates. Mozilla has asked that software users, when visiting an SSL site that uses a certificate issued by one of the DigiNotar CAs, should still be able to override the default trust decisions made by NSS. Because of this, we had to prevent NSS from checking the signature of our knockout certificates. The signatures obviously were no longer correct after our manual modifications.

The easiest way to prevent this was to apply another binary modification to the knockout certificates, in order to make them appear to be self-signed. This means, while several of the original certificates had different fields for subject name and issuer name, we removed the issuer name and inserted another copy of the subject name.

After these modifications, we added the knockout certificates to the
NSSCKBI module and marked them as not trusted.

We have released NSSCKBI version 1.87 which contains a knockout
certificate for the "DigiNotar Root CA" certificate, and 5 knockout
certificates for intermediates. We published a combination of the latest stable release of NSS 3.12.11 with this newer roots module at
ftp://ftp.mozilla.org/pub/mozilla.org/security/nss/releases/NSS_3_12_11_WITH_CKBI_1_87_RTM/

The following is the set of intermediates known to us:

Issuer: C=NL, O=DigiNotar, CN=DigiNotar Root
CA/emailAddress=info@diginotar.nl
Subject: C=NL, O=DigiNotar, CN=DigiNotar Root
CA/emailAddress=info@diginotar.nl
Serial Number: 0c:76:da:9c:91:0c:4e:2c:9e:fe:15:d0:58:93:3c:4c
Not Before: May 16 17:19:36 2007 GMT
Not After : Mar 31 18:19:21 2025 GMT

Issuer: C=US, O=Entrust.net, OU=www.entrust.net/CPS incorp. by ref.
(limits liab.), OU=(c) 1999 Entrust.net Limited, CN=Entrust.net Secure
Server Certification Authority
Subject: C=NL, O=DigiNotar, CN=DigiNotar Services 1024
CA/emailAddress=info@diginotar.nl
Serial Number: 1184640176 (0x469c2cb0)
Not Before: Jul 26 15:59:00 2007 GMT
Not After : Aug 26 16:29:00 2013 GMT

Issuer: C=US, O=GTE Corporation, OU=GTE CyberTrust Solutions, Inc.,
CN=GTE CyberTrust Global Root
Subject: C=NL, O=DigiNotar, CN=DigiNotar Cyber
CA/emailAddress=info@diginotar.nl
Serial Number: 120000525 (0x727100d)
Not Before: Oct 4 10:54:11 2006 GMT
Not After : Oct 4 10:53:11 2011 GMT

Issuer: C=US, O=GTE Corporation, OU=GTE CyberTrust Solutions, Inc.,
CN=GTE CyberTrust Global Root
Subject: C=NL, O=DigiNotar, CN=DigiNotar Cyber CA
Serial Number: 120000505 (0x7270ff9)
Not Before: Sep 20 09:45:32 2006 GMT
Not After : Sep 20 09:44:06 2013 GMT

Issuer: C=US, O=GTE Corporation, OU=GTE CyberTrust Solutions, Inc.,
CN=GTE CyberTrust Global Root
Subject: C=NL, O=DigiNotar, CN=DigiNotar Cyber CA
Serial Number: 120000515 (0x7271003)
Not Before: Sep 27 10:53:32 2006 GMT
Not After : Sep 27 10:52:30 2011 GMT

Issuer: C=NL, O=Staat der Nederlanden, CN=Staat der Nederlanden Overheid CA
Subject: C=NL, O=DigiNotar B.V., CN=DigiNotar PKIoverheid CA Overheid en
Bedrijven
Serial Number: 20015536 (0x13169b0)
Not Before: Jul 5 08:42:07 2007 GMT
Not After : Jul 27 08:39:46 2015 GMT

Issuer: C=NL, O=Staat der Nederlanden, CN=Staat der Nederlanden
Organisatie CA - G2
Subject: C=NL, O=DigiNotar B.V., CN=DigiNotar PKIoverheid CA Organisatie
- G2
Serial Number: 20001983 (0x13134bf)
Not Before: May 12 08:51:38 2010 GMT
Not After : Mar 23 09:50:04 2020 GMT

And what follows are the details of the knockout certificates we have
created. We believe this smaller list is sufficient to handle all the
intermediates listed above, because some of them have identical subject
names.

Issuer: C=NL, O=DigiNotar, CN=DigiNotar Root
CA/emailAddress=info@diginotar.nl
Subject: C=NL, O=DigiNotar, CN=DigiNotar Root
CA/emailAddress=info@diginotar.nl
Serial Number: 0f:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff
Not Before: Jul 27 17:19:37 2007 GMT
Not After : Mar 31 18:19:22 2025 GMT

Subject: C=NL, O=DigiNotar, CN=DigiNotar Services 1024
CA/emailAddress=info@diginotar.nl
Issuer: C=NL, O=DigiNotar, CN=DigiNotar Services 1024
CA/emailAddress=info@diginotar.nl
Serial Number: 268435455 (0xfffffff)
Not Before: Jul 26 15:59:01 2007 GMT
Not After : Aug 26 16:29:01 2013 GMT

Subject: C=NL, O=DigiNotar, CN=DigiNotar Cyber
CA/emailAddress=info@diginotar.nl
Issuer: C=NL, O=DigiNotar, CN=DigiNotar Cyber
CA/emailAddress=info@diginotar.nl
Serial Number: 268435455 (0xfffffff)
Not Before: Oct 4 10:54:12 2006 GMT
Not After : Oct 4 10:53:12 2011 GMT

Subject: C=NL, O=DigiNotar, CN=DigiNotar Cyber CA
Issuer: C=NL, O=DigiNotar, CN=DigiNotar Cyber CA
Serial Number: 268435455 (0xfffffff)
Not Before: Sep 27 10:53:53 2006 GMT
Not After : Sep 20 09:44:07 2013 GMT

Subject: C=NL, O=DigiNotar B.V., CN=DigiNotar PKIoverheid CA Overheid en
Bedrijven
Issuer: C=NL, O=DigiNotar B.V., CN=DigiNotar PKIoverheid CA Overheid en
Bedrijven
Serial Number: 268435455 (0xfffffff)
Not Before: Jul 5 08:42:08 2007 GMT
Not After : Jul 27 08:39:47 2015 GMT

Subject: C=NL, O=DigiNotar B.V., CN=DigiNotar PKIoverheid CA Organisatie
- G2
Issuer: C=NL, O=DigiNotar B.V., CN=DigiNotar PKIoverheid CA Organisatie - G2
Serial Number: 268435455 (0xfffffff)
Not Before: May 12 08:51:39 2010 GMT
Not After : Mar 23 09:50:05 2020 GMT

If you have copies of additional intermediates that you would like to
see blocked, please send us full copies of the certificates, and we will see if further action is necessary.

Regards
Kai Engert

In November 2009 a Man-In-the-Middle vulnerability for SSL/TLS/https was made public (CVE-2009-3555), and shortly afterwards demonstrated to be exploitable. In February 2010 researchers published RFC 5746 that described how servers and clients can be made immune. Software that implements the TLS protocol enhancements became available shortly afterwards. Most modern web browsers are patched, but the solution requires that both browser developers and website operators take action.

Unfortunately, 16 months later, many major websites, including several ones that deal with real world transactions of goods and money, still haven't upgraded their systems.

Even worse, for a big portion of those sites it can be shown that their operators failed to apply the essential configuration hotfix. They support the style of handshakes that can allow a MITM attacker to inject attack data into the transaction stream.

Here is a list of patched and unpatched popular sites, along with more background information. The patched sites demonstrate that patching is indeed possible.

Given that attackers could execute malicious transactions with a customer's credentials, customers should demand that this security issue gets resolved quickly. What can we do to remind service providers that fixing this issue deserves a high priority?

This is a report about a Man-In-The-Middle (MITM) situation that I experienced a couple of days ago while travelling.

I spent the night in a hotel in Warsaw, Poland. I bought access to the Internet WiFi. When my Thunderbird email client started, it automatically checked my various email accounts.

For the SSL connection to imap.googlemail.com (also known as imap.gmail.com) at port 993, Thunderbird warned me about a certificate error. The certificate presented by IP address 74.125.115.16 was issued to

Subject: C=US, ST=California, L=Mountain View, O=Google Inc, CN=imap.googlemail.com

and it was issued by

Issuer: C=US, ST=California, L=Sunnyvale, O=Fortinet, OU=Certificate Authority, CN=FortiGate CA/emailAddress=support@fortinet.com


Given that Google's computers usually use certificates from Certificate Authorities that are trusted by Thunderbird, and given that this particular is not, this appeared to be a MITM attack.

The next question was, who is it that sends this unknown certificate to me? Is it sent by Google's IP address 74.125.115.16, or is it sent to me by another computer that is physically present on the route between me and Google?

I started an analysis. I made remote connections to other computers, including my server located in Germany. I repeated the connection attempt from my server to the same IP 74.125.115.16. For this connection, things looked good, I received a valid certificate from a CA trusted by Thunderbird. A friend of mine, who lives in Warsaw, repeated the test using his home Internet connection. Again, things looked good.

The conclusion is, someone along the route between me (sitting in the hotel) and the imap.google.com server, was using a MITM software or hardware to hijack my connection.

Luckily I noticed it, thanks to the automatic checks used by Mozilla Thunderbird.

So, who is it that performed the attack on me? That's not easy to tell.

Without blaming anyone, I'm naming the involved parties:
- I was in the Metropol Hotel Warsaw
- the Internet provider appears to be atman.pl
- anyone else in control of one of IP addresses on the route between the hotel and Google:

$ traceroute 74.125.115.16
traceroute to 74.125.115.16 (74.125.115.16), 30 hops max, 60 byte packets
 1  192.168.69.254 (192.168.69.254)  4.169 ms  4.206 ms  4.287 ms
 2  * * *
 3  z-atman.48.warsawhotelmetropol.com (213.189.36.165)  15.412 ms  15.578 ms  15.712 ms
 4  r7-global-lt-3-2-0-30.atman.pl (85.232.232.13)  15.903 ms  16.043 ms  16.239 ms
 5  72.14.219.114 (72.14.219.114)  38.372 ms  38.978 ms  39.663 ms
 6  209.85.240.64 (209.85.240.64)  40.449 ms 209.85.241.110 (209.85.241.110)  30.391 ms 209.85.240.64 (209.85.240.64)  40.327 ms
 7  209.85.248.94 (209.85.248.94)  90.287 ms  65.451 ms 72.14.233.104 (72.14.233.104)  43.865 ms
 8  72.14.239.94 (72.14.239.94)  134.331 ms  134.560 ms 216.239.43.90 (216.239.43.90)  124.102 ms
 9  209.85.243.114 (209.85.243.114)  139.958 ms  140.091 ms 209.85.241.222 (209.85.241.222)  136.167 ms
10  209.85.241.207 (209.85.241.207)  140.229 ms 209.85.251.228 (209.85.251.228)  140.453 ms 64.233.174.87 (64.233.174.87)  129.356 ms
11  209.85.253.185 (209.85.253.185)  134.299 ms 209.85.253.181 (209.85.253.181)  130.482 ms 209.85.242.181 (209.85.242.181)  127.593 ms
12  vx-in-f16.1e100.net (74.125.115.16)  128.946 ms  130.188 ms  129.702 ms

The first step on identifying the potential attacker was to inspect the certificate that was presented by the MITM. The certificate claimed to be issued by FortiGate CA. What does this mean?

FortiGate is a product sold by Fortinet, Inc. It is a device that can be used to perform a various set of actions on Internet connections. For example, a company could use it to block connections, for example to disallow employees to access their private email. Another use is being able to watch the contents of encrypted connections between a browser and Internet services.

It is debatable whether the use of such appliances is acceptable. There might be scenarios where this is acceptable. For example, one could think of a governmental agency that allows its employees to make private connections to the Internet, but at the same time all employees have agreed to never publish any confidential information, and employees might have agreed that all their actions are being watched while at work, in order to verify that no confidential data is being published. This might be an acceptable scenario.

In my opinion, using such a MITM appliance, for the purpose of hijacking connections of hotel guests to Email service providers, is clearly not acceptable. Other's have to decide whether this is a criminal situation, but I think it might be.

I've talked to a representative of the German office of Fortinet. I was told, by default the FortiGate appliances come with a zero configuration. Any blocking or MITM behaviour of the FortiGate appliance must be deliberately enabled by the customer.

In my understanding, we cannot blame Fortinet. They appear to have provided the tool being used to perform an attack, but it appears that someone else has installed and abused that tool.

Unfortunately I don't know whom to blame. This would have to be decided by an investigation.

The important lesson that we can take away from this story:

Don't lightheartedly ignore security warnings. They can be real attacks. In my scenario, if I had decided to connect anyway, the person performing the MITM would have been able to hijack my password to one of my accounts.

(Also, I have a technical question about the kind of connection used by Android's Google Mail application. Does it use IMAP with SSL on port 993? I hope it does not. When I connected the Android phone to the hotel wifi, it successfully connected. I did not get any security warning. This might be OK, depending on which protocol is being used by the software. If know what it uses, please let me know. Thanks. (To be on the safe side, I had immediately changed my password using a non-MITM client.))

[Update 1] (My separate worry about the Android app was unfounded. Meanwhile I've been in contact with a representative from Google. I've been told that the "Google Mail / Gmail App" on Android always uses https on port 443. This port was not affected by the MITM attack. The attack was limited to imap/ssl on port 993.) [/Update 1]

FYI, I saved the MAC address of the Wireless Access point that I've been connected to, and can provide it to authorities. It looked like 00:4F:62:xx:xx:xx. There were other Access points in the hotel. Unfortunately only this one AP's signal was strong enough to keep a connection to it from my room.

[Update 2] This scenario appears to happen more frequently.

After reading the article, Stephen Davidson found the following two discussions on the web, which appear to report the same issue:
- report 1 (experienced at a hotel, probably located in the USA, because the author mentioned Sprint)
- report 2 (seems to be happening at a school in the US, last comment indicates probably caused by an accidental misconfiguration)

I was able to find some additional reports:
- report 3 - report 4 - report 5 - report 6 - report 7 (search that page for "imap.gmail")

There's one detail that surprises me most: I am able to find several reports about MITM for imap.gmail.com mentioning a FortiGate CA, but I cannot find reports mentioning Fortigate and IMAP in combination with some other imap server. Is the explanation that Gmail is the most popular imap service on the web, and that users of other imap servers are aware of the fact that they are being monitored by their environment? [/Update 2]

[Update 3] Clarification: The MITM experience was only seen in one combination of host and port (out of the combinations I tried). In my opinion, if the box accidentally ran an incorrect configuration by mistake, I would have expected it to equally affect all connections on a set of port numbers. However, access to other IMAP/SSL hosts on port 993 (including my own mail server) were NOT affected and used trusted certs. The MITM appears to have been specifically targeted to a certain (set of) host(s). [/Update 3]

    MITM used this cert:
    Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number:
                4c:9f:a7:fd:00:00:00:48
            Signature Algorithm: sha1WithRSAEncryption
            Issuer: C=US, ST=California, L=Sunnyvale, O=Fortinet, OU=Certificate Authority, CN=FortiGate CA/emailAddress=support@fortinet.com
            Validity
                Not Before: Apr 22 20:17:57 2010 GMT
                Not After : Apr 22 20:27:57 2011 GMT
            Subject: C=US, ST=California, L=Mountain View, O=Google Inc, CN=imap.googlemail.com
            Subject Public Key Info:
                Public Key Algorithm: rsaEncryption
                    Public-Key: (1024 bit)
                    Modulus:
                        00:ad:81:6b:dc:b4:15:cf:1e:b6:f3:bf:d1:39:7d:
                        42:26:77:23:55:29:28:05:61:1e:ea:bb:7b:a2:02:
                        d1:10:b3:9e:49:d9:b4:8b:54:e5:50:78:0d:52:e1:
                        43:ad:d9:53:fd:2f:a5:23:56:aa:36:7f:54:88:1c:
                        06:68:96:18:dc:b7:43:5e:f3:9a:b6:bc:a8:75:d4:
                        fd:e0:c4:7f:65:3b:8e:63:1f:bf:6e:f4:f5:c1:de:
                        f5:58:f5:9a:c9:6f:2d:a9:75:b7:d4:04:97:3a:d9:
                        f4:22:1a:57:bd:26:36:2c:52:64:2c:b8:77:45:c2:
                        0c:1e:80:72:e9:d4:25:8e:59
                    Exponent: 65537 (0x10001)
        Signature Algorithm: sha1WithRSAEncryption
            6f:e7:95:81:3b:a6:99:6b:35:e2:48:09:77:63:d1:f6:7c:1c:
            bb:5c:67:60:5d:00:ac:2d:7c:2f:8b:02:98:75:5c:1e:43:4b:
            f4:de:64:6e:9d:c6:23:5b:ee:ce:23:13:e5:29:cd:50:b9:47:
            ae:dd:c9:8f:82:3a:7a:3d:cc:f0:35:33:ea:bd:25:6f:02:89:
            e8:3d:27:b7:ee:b7:6f:95:21:a1:a2:6c:4b:2e:17:f8:eb:f1:
            dd:96:90:0e:32:ee:80:f9:61:79:c6:88:1e:75:29:e1:a8:d1:
            34:58:b0:a6:d7:c7:99:18:89:57:8d:b7:f5:c0:73:d6:ae:66:
            a4:e6:af:ce:85:71:e1:de:89:8f:e9:29:65:e5:f3:75:23:02:
            9e:9d:a6:8c:00:d7:9a:f9:7f:13:61:b6:79:65:1e:50:b0:8e:
            dd:bf:cd:b8:7b:72:c2:fa:a3:ab:a0:e1:ca:6f:54:8d:79:37:
            43:0e:98:7a:b0:d6:3e:f9:b5:82:e2:89:e3:5b:65:c2:ec:3a:
            cf:ac:1b:f2:b8:c6:16:74:6c:2b:c4:ce:42:60:e2:93:23:1e:
            13:d5:54:f2:6a:c2:93:94:91:8f:02:6e:1f:30:8d:29:2b:06:
            4b:0f:0a:8d:92:16:ad:fc:e2:73:7c:66:0e:24:14:5a:fd:5b:
            54:42:63:71
-----BEGIN CERTIFICATE-----
MIIDDzCCAfegAwIBAgIITJ+n/QAAAEgwDQYJKoZIhvcNAQEFBQAwgaUxCzAJBgNV
BAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRIwEAYDVQQHEwlTdW5ueXZhbGUx                                                                                                                               
ETAPBgNVBAoTCEZvcnRpbmV0MR4wHAYDVQQLExVDZXJ0aWZpY2F0ZSBBdXRob3Jp                                                                                                                               
dHkxFTATBgNVBAMTDEZvcnRpR2F0ZSBDQTEjMCEGCSqGSIb3DQEJARYUc3VwcG9y
dEBmb3J0aW5ldC5jb20wHhcNMTAwNDIyMjAxNzU3WhcNMTEwNDIyMjAyNzU3WjBt
MQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTEWMBQGA1UEBxMNTW91
bnRhaW4gVmlldzETMBEGA1UEChMKR29vZ2xlIEluYzEcMBoGA1UEAxMTaW1hcC5n
b29nbGVtYWlsLmNvbTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEArYFr3LQV
zx6287/ROX1CJncjVSkoBWEe6rt7ogLRELOeSdm0i1TlUHgNUuFDrdlT/S+lI1aq
Nn9UiBwGaJYY3LdDXvOatryoddT94MR/ZTuOYx+/bvT1wd71WPWayW8tqXW31ASX
Otn0IhpXvSY2LFJkLLh3RcIMHoBy6dQljlkCAwEAATANBgkqhkiG9w0BAQUFAAOC
AQEAb+eVgTummWs14kgJd2PR9nwcu1xnYF0ArC18L4sCmHVcHkNL9N5kbp3GI1vu
ziMT5SnNULlHrt3Jj4I6ej3M8DUz6r0lbwKJ6D0nt+63b5UhoaJsSy4X+Ovx3ZaQ
DjLugPlhecaIHnUp4ajRNFiwptfHmRiJV4239cBz1q5mpOavzoVx4d6Jj+kpZeXz
dSMCnp2mjADXmvl/E2G2eWUeULCO3b/NuHtywvqjq6Dhym9UjXk3Qw6YerDWPvm1
guKJ41tlwuw6z6wb8rjGFnRsK8TOQmDikyMeE9VU8mrCk5SRjwJuHzCNKSsGSw8K
jZIWrfzic3xmDiQUWv1bVEJjcQ==
-----END CERTIFICATE-----