DigiNotar Removal Follow Up

Johnathan Nightingale

70

Earlier this week we revoked our trust in the DigiNotar certificate authority from all Mozilla software. This is not a temporary suspension, it is a complete removal from our trusted root program. Complete revocation of trust is a decision we treat with careful consideration, and employ as a last resort.

Three central issues informed our decision:

1) Failure to notify. DigiNotar detected and revoked some of the fraudulent certificates 6 weeks ago without notifying Mozilla. This is particularly troubling since some of the certificates were issued for our own addons.mozilla.org domain.

2) The scope of the breach remains unknown. While we were initially informed by Google that a fraudulent *.google.com certificate had been issued, DigiNotar eventually confirmed that more than 200 certificates had been issued against more than 20 different domains. We now know that the attackers also issued certificates from another of DigiNotar’s intermediate certificates without proper logging. It is therefore impossible for us to know how many fraudulent certificates exist, or which sites are targeted.

3) The attack is not theoretical. We have received multiple reports of these certificates being used in the wild.

Mozilla has a strong history of working with CAs to address shared technical challenges, as well as responding to and containing breaches when they do arise. In an incident earlier this year we worked with Comodo to block a set of mis-issued certificates that were detected, contained, and reported to us immediately. In DigiNotar’s case, by contrast, we have no confidence that the problem had been contained. Furthermore, their failure to notify leaves us deeply concerned about our ability to protect our users from future breaches.

Staat der Nederlanden Certificates

DigiNotar issues certificates as part of the Dutch government’s PKIoverheid (PKIgovernment) program. These certificates are issued from a different DigiNotar-controlled intermediate, and chain up to the Dutch government CA (Staat der Nederlanden). The Dutch government’s Computer Emergency Response Team (GovCERT) indicated that these certificates are issued independently of DigiNotar’s other processes and that, in their assessment, these had not been compromised. The Dutch government therefore requested that we exempt these certificates from the removal of trust, which we agreed to do in our initial security update early this week.

The Dutch government has since audited DigiNotar’s performance and rescinded this assessment. We are now removing the exemption for these certificates, meaning that all DigiNotar certificates will be untrusted by Mozilla products. We understand that other browser vendors are making similar changes. We’re also working with our Dutch localizers and the Bits of Freedom group in the Netherlands to contact individual site operators using affected certificates (based on the EFF’s SSL Observatory data).

The integrity of the SSL system cannot be maintained in secrecy. Incidents like this one demonstrate the need for active, immediate and comprehensive communication between CAs and software vendors to keep our collective users safe online.

Johnathan Nightingale
Director of Firefox Engineering

70 responses

  1. Michichael wrote on :

    A well made point. Realistically, 90% of people just click “ignore” anyway when they encounter a certificate error, but the rest of us might get suspicious when we see bad certs. The entire system needs to be re-evaluated, but that can’t happen when there are bad root CA’s.

  2. Ferdinand wrote on :

    Well done, giving us a quick and clear resolution. As it turned out this was one useless CA. The Dutch government has some things left to clarify to its citizens.

    I hope Mozilla will consider the possible implications beyond the immediate scope of Diginotar. This episode really puts a dent in the chain of trust.

  3. ocrete wrote on :

    Knowing that these certificates are actively bad, would it be possible to make Firefox refuse to load any site that uses them? And not even give the user the option to accept it, I saw that you do that on expired certs now.

  4. biscuit town wrote on :

    Everyone has known that SSL has been broken for years because of how “trust” is managed in browsers nowadays but, as usual, nobody cares until we are certain someone has already been screwed.

  5. d2 wrote on :

    Bravo, Jonathan. SSL needs to have something other than lowest price guiding corporate purchase decisions, and while there’ll be backsplatter from any firm stuck with DigiNotar, this NEEDS to be done.

    2 thoughts:

    – What about Comodo? Did I miss your revocation of their CA? If not, WHY not?

    – Is Moxie’s notary mechanism ( Convergence ) or CMU’s (Perspectives?) on your radar for Firefox?

  6. Daniel Veditz wrote on :

    The sad thing is you can’t even blame race-to-the-bottom pricing in this case, the DigiNotar certs were about a thousand euros.

    I’m intrigued by the Perspectives/Convergence experiments and we’re definitely watching to see how they work in practice and at scale. We have no plans to build either one into Firefox at this time.

  7. ldpreload wrote on :

    d2: Comodo made an honest mistake and handled it well. Revoking it out of retribution would do nothing — Comodo is one of the few CAs we know to be able to handle mistakes well! We have no idea whether most of the CAs have made mistakes, let alone how they would handle it. (As the current incident shows, “poorly” is a good guess.)

    Comodo is also a sufficiently big player that permanently revoking it would yield a lot of individual employees moving elsewhere in the industry. Since ultimately all failures are human, that would just decrease accountability.

    And finally, revoking Comodo’s CA would just teach CAs that, no matter how well they recover, they’re in trouble if a mistakeis public, so better to just never publicize mistakes.

  8. Gervase Markham wrote on ::

    ocrete: the block is non-overridable for certificates issued after the 1st of July 2011.

  9. Fredrick Rybarczyk wrote on :

    DNSSEC? Love, @fredrick_r

  10. Doublehypocrite wrote on :

    You still have trust comodo. Why?

  11. Martin wrote on :

    Michichael, there is no technical prohibition against working on parallel infrastructures for modelling trust on the web, including the well-known ultimately flawed, unlimited limitless trust CA anchor model. Start coding ;)

  12. Ramón Antonio wrote on ::

    Don’t know what is the policy to select CA but here in Spain we are always missing the Fabrica Nacional de Moneda y Timbre (Spanish Royal Mint) as a CA. It issues certificates for all governmental websites and personal/company certificates. We always have to add it manually for electronic proceedings and most users don’t know how to do it.

  13. anonymous wrote on :

    Comodo immediately reported the incident. It seems DigiNotar didn’t.

  14. SteveL wrote on :

    Can’t we have a way of blocking all Certs issued within a date range? So we could block Diginotar’s certs from July 1 until Dec 31 (i.e. not until somebody else had audited the system and felt it was secure). Then people like Comodo and other too-big-to-fail CAs could be partially blocked as punishment for issuing any invalid certificates

  15. alex wrote on :

    I’ll admit a definite ignorance about CA’s and such, but I’m curious as to why CRL’s aren’t more fully employed here. Isn’t a CRL a list of revoked certificates that the browser can check and then set appropriate trust attributes based on any certs listed? Had DigiNotar issued a CRL for the bad certs and the browser had it stored in the cert database, would this not be sufficient to warn users that they are accessing a site using one of the bogus certs? And where does OSCP fit in? I’ve got it enabled, but would it even help in this case?

    It just seems ludicrous that users have to update their applications with a hardcoded blacklist of cert’s and CA’s everytime there is a compromise to the SSL system published in the news.

  16. Ken Dawber wrote on ::

    Judge, Jury and Executioner.

    Millions of people died in various wars to stop this type of thing happening. The issue isn’t whether DigiNotar did the the right thing or not. It is that we cannot have a judge, jury and executioner all as the same organization and all of this without fair trial and due process etc.

    The fact that mozilla can act as all three shows something very wrong with the process.

    This is particularly important in this case as there are large numbers of innocent third parties who have certificates from DigiNotar. One person is accused of doing something wrong so we go kill kill kill anybody and everybody associated with them.

    Due process should have external parties acting as judge and jury.

    In the period that was required to set this up, there obviously needs to be some safeguards but I doubt if the kill of the full root in needed for that. Since the issue here is DigiNotar’s competence as a root, a minimum needed is to try to allow another competent authority take temporary control while the issue is being sorted.

  17. Jasem wrote on :

    I should say even after installing 6.0.1 update and restarting the browser, I noticed that DigiNotar still exists in Certificate Manager. I don’t know why but I had to delete it manually.

  18. jmdesp wrote on :

    @Martin : It’s definitively not about coding. It’s all in conceiving a trust model that really handles failure better. For this, it’d be good to realize that inventing a completely new format is not really required, it means a lot of library rewriting for little gain, the part of “I devise new tools” is a strong diversion from the actual aim. If you’re smart you can keep the X509 format and completely change the trust model and it helps you focus on changes that truly alter the trust model.

    The systems based on distributed trust seem to me to be the nearer from really adding something new and significant, but they rise some serious anonymity issues.
    Google’s CA pinning has now demonstrated that it actually brings something useful, and I think Mozilla should very seriously think about integrating it.

  19. Blah wrote on :

    ldpreload, d2:
    IIRC, Comodo did not simply make a mistake and then correct it… rather, Comodo wrongfully issued a cert for a Mozilla domain, claimed to have corrected what allowed the wrongful issuance to occur, and then much later wrongfully issued certs again. The wrongful issuance would have been prevented had Comodo made the corrections it claimed it made the first time around.

    Comodo’s roots should be permanently yanked.

  20. muddybulldog wrote on :

    It’s not about punishing, it’s about preserving the system as best as possible. Comodo made efforts to work with those impacted and the browser vendors to get their mistake rectified. As was pointed out, punishing registrars is just going to cause coverups making things worse, not better.

  21. Dave wrote on :

    >The sad thing is you can’t even blame race-to-the-bottom pricing
    >in this case, the DigiNotar certs were about a thousand euros.

    Diginotar had been issued a monopoly license to print money by the Dutch government. It’s surprising they didn’t charge a million euros per cert, since there was no-one else you could go to.

  22. Thaddy wrote on ::

    As a Dutchman and as a security professional I really appreciate this.
    I suspect this was always the deal: hardcoded exceptions is pollution.
    Our government was sensible enough to agree and just needed a couple of days to do the assessments. Am I right?
    I think this goes to show that at least our government is not afraid to take extreme and never seen before action at short notice.
    Now, how did Comodo got away with almost the same?

  23. Jeroen wrote on :

    When will the update be ready? Dutch citizens communicating with the government have been vulnerable since July 10 — it is confirmed that the PKI Overheid chains were also hacked — they need security warnings ASAP.

  24. colfer wrote on :

    Why is the July 1 exemption still in the code?

  25. Daniel Veditz wrote on :

    Jeroen: We’re currently testing the fix and plan to release Tuesday morning. If people are impatient to get the fix they can download an “Aurora” build from http://www.mozilla.org/en-US/firefox/channel/ or download the candidates themselves from

    https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.6.22-candidates/build2/

    https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/6.0.2-candidates/build2/

  26. Daniel Veditz wrote on :

    jmdesp: we’re considering cert pinning but would prefer a distributed model to Google’s hardcoding–hardcoding doesn’t scale and not very open. Still, it clearly works and might be better than nothing in the short-term.

  27. piet wrote on :

    Too bad the 6.0.1 update has the “Staat der Nederlanden” exception built-in. I can’t untrust the diginotar cert that’s subordinate to “Staat der Nederlanden” manually either it seems.

  28. Daniel Veditz wrote on :

    Ken Dawber: a “competent party” taking over DigiNotar’s root and responsibilities doesn’t put the genie back in the bottle — an unknown number of fraudulent certificates are out there and being used and not enough is known to block just those certificates. It is unfortunate this action impacts other innocent DigiNotar customers but it is the only remedy. DigiNotar’s failing puts everyone on the web at risk, many multiples more people than are harmed by yanking DigiNotar.

    “Judge, Jury and Executioner” doesn’t apply.DigiNotar does not have a right to a trusted root in Mozilla products, and Mozilla’s action only affects Mozilla’s users. DigiNotar is free to continue using their certificate and users who think we’ve made the wrong call are free to use another browser.

  29. Thaddy wrote on ::

    Staat der Nerderlanden is also withdrawn. Jeroen: ITYS
    Good and sensible handling by all parties, btw.

  30. Name wrote on :

    This blog post does not explain why Mozilla gave credulence to the Dutch government’s assertions that the PKIoverheid program was safe. Especially in light of the revelation that the Dutch government itself concluded that this assertion was false, I think this warrants a little more commentary on Mozilla’s behest.

  31. Bob Relyea wrote on :

    re: datebase revocation.

    That’s an idea that we are looking at implementing. The Comodo case showed holes in our ability to knock out specific certificates. Patches for some of these issues are still working their way in our system. DigiNotar has now shown other holes. We’ve handled both of these cases initially way up in the PSM processing of of SSL certificates because it provided us with the broadest coverage for the quickest time. Patches would handle some of these issues are making their way through the system. As they get applied, additional patches will help fill in the holes.

    Date based revocation (where the valid dates a leaf cert can have is constrained by the CA) is definately in the cards. In addition, we may make the constraint on a public key or String in the DN as well.

  32. Thaddy wrote on ::

    IMNSHO opinion: the Comodo case still sucks. I think the Dutch government did an unprecedented step to acknowledge the insecurity of their certificates. And in only three days (this is a government, mind you)

    This is a GOOD thing for the internet, just as the Comodo case is a BAD thing.

    And open source proved its strength, if I would have read the hardcoded snippets about the exclusion/inclusion of certain certificate series and it was April fool’s day you “wouldn’t have fooled me”…

  33. thaddy wrote on ::

    Jeroen, Piet: the security warnings are mostly in place, I am surprised to say. But if Donner says so, somebody listens and the rest falls to sleep (Dutch Joke):
    For the non – Dutch: The Dutch government withdrew all certificates of DigiNotar AND all public ones issued by itself and most of the affected sites indeed show an appropiate warning.
    Some rather big municipalities DO NOT COMPLY YET however, exposing a potential 4 mil. (as of saterday 23:00 CET).

    And the government has given ample warning internally the last few days and externally from yesterday. They also withdrew all statements that anything was safe. The reversed it: nothing is safe, unless.

    Which it seems is in a large part due to temporary patchwork from the side of the internet community as a whole.
    All in all: seems to be handled in a sensible way. I really can’t imagine a better short term solution than seems to be executed in this case.

    We are experiencing/living history here, nobody get’s that?

  34. Boris wrote on :

    Jasem, it’s still in the Cert store so it can be marked explicitly untrusted without possibility of override. Your removing it makes it like any other unknown certificate (that is, gives you the option to trust it when you see it). As in, it makes you _less_ secure.

  35. Alex Bishop wrote on :

    Ramón Antonio said: “Don’t know what is the policy to select CA”

    It’s described at: http://www.mozilla.org/projects/security/certs/policy/

  36. colfer wrote on :

    The date exemption for being able to override is still in the code:

    security/manager/ssl/src/nsNSSCallbacks.cpp on mozilla-release:
    1043 // If any involved cert was issued by DigiNotar,
    1044 // and serverCert was issued after 01-JUL-2011,
    1045 // then worsen the error to revoked.

    Aside from there being no evidence July 1 is a safe cut-off, can the dates be forged?

  37. Metasansana wrote on :

    My my, isn’t it an exciting time to be into Internet security?

  38. rbdg wrote on :

    Echoing what @jasem said earlier. Installed v6.0.1 @ 0500Hr (GMT+8) today and found DigiNotar STILL listed as trusted CA.

    The worst part of it all is that I was running v5.0 before the update and had ALREADY REMOVED DigiNotar MANUALLY.

    QUESTION for JN:
    Does it mean that FF minor/major version updates/upgrades have/will routinely ignored/ignore custom CA additions/removals and have/will always overwritten/overwrite the customized CA list???

    Been a FF fan for years, but maybe it’s time to ponder switching over to Chrome fulltime? *SIGH*

  39. Hay wrote on ::

    Hi everyone,
    one of thing that isn’t clear to me after reading all the news on the revoking of the root Staat der Nederlanden CA certificate is what will happen to older browsers. It is a well known fact that users are slow in upgrading their older browsers, so what will happen if a user with an older browser visits a site using the affected root CA? And if the Staat der Nederlanden releases a new certificate how do they get that in their browser? I’ve read some stuff about Windows automatically updating certificates with Windows Update, but i’m not sure if that also affects Firefox and other browsers.

  40. Robert wrote on :

    Fox-IT has just released an interim report on the DigiNotar breach.
    Available at http://tweakimg.net/files/upload/Operation+Black+Tulip+v1.0.pdf

  41. Dave wrote on :

    @Hay: It’s as simple as you think it is. If people don’t install the security updates for their browser, then they won’t get any security updates for their browser. This incident adds to the litany of reasons to scream at people who ignore, or sometimes actively reject, updates.

  42. David Bernier wrote on :

    I support Mozilla’s decision because of
    the risks and potential for trouble if
    nothing were done about DigiNotar-issued
    certificates…

  43. David Bernier wrote on :

    @Hay:

    I think hopefully that either DigiNotar or the
    Dutch government CA can revoke the
    potentially dubious Staat der Nederlanden Certificates.

    As for old browsers, I believe New additional
    root CA certificates can be imported into
    the browser by the user. But it’s a
    highly technical issue, so I’m not sure it’s
    the best idea for people with old browsers;
    it might be better to simply recommend
    upgrading or changing to a up-to-date
    browser & version.

  44. Ingo Kresse wrote on :

    My, my, my, as if it couldn’t get worse: The interim report of Fox-IT (the company investigating in DigiNotar) reveals that all CA servers were in the same windows domain, giving a domain administrator access to _every_ CA server.

    http://www.diginotar.nl/Portals/7/Persberichten/Operation%20Black%20Tulip%20v1.0a.pdf

    They knew that one of their servers could be compromised (since they received a fraudulent certificate issued by them) when they made their press release, stating that their PKIoverheid CA servers were ‘unaffected’.

    http://www.vasco.com/company/press_room/news_archive/2011/news_diginotar_reports_security_incident.aspx

    A horrible thought that we trusted them…

  45. Daniel Veditz wrote on :

    colfer asks:
    > Aside from there being no evidence July 1 is a safe cut-off, can
    > the dates be forged?

    There is -some- evidence that July 1 is OK. Earlier (when we didn’t know about the evidence of hacking in June) July 10 was the earliest discovered certificate. Later it was revealed some hacking was discovered as early as June 17, but with more detailed log forensics it still looks like July 2 was the first cert-issuing attempt and July 10 was the first success.

    Depending on the kind of access dates in certificates can potentially be forged. In the current attack it looks like certificate dates will be reliable, but if the attacker created a signing certificate rather than a server certificate then then they could definitely forge the dates. Current evidence says the attacker didn’t get one of these powerful certs. Then again, the first look missed a whole batch of things so the second might have as well.

  46. Daniel Veditz wrote on :

    @rbdg: You cannot “remove” the built-in certs because they are, well, built-in. If you press the “delete or distrust” button, though, the certificate –IS– marked as untrusted.

    The 6.0.1 update still includes -a- DigiNotar certificate but it is marked as untrusted. For technical reasons it’s better to have an explicit distrust than have it be simply “unknown”.

    Any certificate changes you make (overrides, marking roots as untrusted) are saved in your profile. These changes are NEVER overwritten by an update.

  47. jmdesp wrote on :

    @Daniel : I think it’s a know bug that you can make it appear in the GUI that the cert is no more there, but that it will be, visually, reinstated when you update, maybe as untrusted.

  48. petr_p wrote on :

    @Ramón Antonio: Mozilla overlooks all these national CAs because they are relatively small and because they focus on higher level than Mozilla (Mozilla is interested in commercial web server or MIME usage that is about money only, while national CAs target on qualified usage that puts legal binding replacing hand-written signatures).

    Effectively qualified CAs that demand much higher security standards (e.g. you need to prove you identity in person by showing two state-issued identity cards) are not trusted by Mozilla, whereas CAs that issue certificate just by vague e-email or HTTP challenge/response are trusted by Mozilla. What a sad reality.

  49. gebruiker wrote on :

    in the certificate manager there are still Diginotar Certificate names and what about the problems mentioned here is the the same issue?

    Add Security Exception

    You are about to override how Thunderbird identifies this site.
    Legitimate banks, stores, and other public sites will not ask you to do this.

    Server
    Location: pop3.live.com:995

    Certificate Status
    This site attempts to identify itself with invalid information.

    Wrong Site
    Certificate belongs to a different site, which could indicate an identity theft.

    example:
    http://forums.mozillazine.org/viewtopic.php?f=39&t=2239733&p=10968653

  50. Nima wrote on :

    That’s the first step. We need a new design in security.

More comments:1 2