Introducing Masche: memory scanning for server security

winterOfSecurity_logo_dark_vertical2Mozilla operates thousands of servers to build products and run services for our users. Keeping these servers secure is the primary concern of the Operations Security team, and the reason why we have built Mozilla InvestiGator (MIG), a cross-platform endpoint security system.

MIG can inspect the file system and network information of thousands of hosts in parallel, which greatly helps increase visibility across the infrastructure. But until recently, it lacked the ability to look into the memory of running processes, a need that often arises during security investigations.

This is where Mozilla Winter of Security team Masche comes into play. Over the last 6 months, students Marco Vanotti, Patricio Palladino, Nahuel Lascano and Agustin Martinez Suñé from University of Buenos Aires, Argentina, have designed and built a memory forensics library that runs on Linux, Mac OS and Windows.

Masche provides basic primitives for scanning the memory of processes without disrupting the normal operations of a system. Compared with frameworks like Volatility or Rekall, Masche does not provide the same level of advanced forensics features. Instead, it focuses on searching for regexes and byte strings in the processes of large pools of systems, and does so live and very fast.

The source code of Masche is completely open source under the Mozilla Public License, version 2.0, and can be found on github at

The effort needed to implement a complex scanning solution across three operating systems, and complete this work in just a few months, was no easy feat. In the recording below, Marco, Patricio, Nahuel and Agustin present their work, the challenges they had to overcome, and the necessity to mix Go and C to access the lowest parts of the system.

We are now integrating Masche as a module for MIG with the goal to deploy it across our infrastructure. As we use it more for live memory forensics, we will continue to improve its scanning capabilities and contribute the results back to the community.

Revoking Intermediate Certificates: Introducing OneCRL

Users of Firefox from Firefox 37 will be protected by a new feature called OneCRL. This is a new mechanism we have introduced to push lists of revoked intermediate certificates to the browser.

Using OCSP for certificate revocation doesn’t serve users very well. For online revocation checks, either you have a system that fails open or you accept the performance penalty of checks that are more strict (as is the case for EV certificates). OCSP stapling can remove the need for live revocation checks, but currently, only only around 9% of TLS connections use it.

OneCRL helps speed up revocation checking by maintaining a centralized list of revoked certificates and pushing it out to browsers. Currently, if a serious incident occurs that requires certificates to be revoked, we release an update to Firefox to address the problem. This is slow because it takes some time for users to get the security update and restart their browsers. There’s also cost involved in producing an update (and users downloading it).

Firefox already has a mechanism for periodically checking for things that may harm users called blocklisting. OneCRL extends the blocklist to include certificates which should be revoked in addition to the errant add-ons, plugins and buggy graphics drivers currently included. This lets users get the benefit of fresh revocation information without having to update or restart their browser.

The other major benefit of OneCRL is speed. For certificates covered by OneCRL, there is no need to do live OCSP checks, so revocation checking incurs no additional latency. The is especially important for EV certificates, where a positive OCSP response is required.

Right now, OneCRL only covers CA intermediate certificates (in order to limit the size of the blocklist). OneCRL is updated when a CA in Mozilla’s root program notifies Mozilla that an intermediate certificate needs to be revoked.

The initial version of OneCRL that we have today is an important step. It will speed up revocation checking, especially for sites that use EV certificates. But we’re not done yet. We’re working on scaling up OneCRL so that its benefits apply more broadly, and on automating the collection of revocation information so that it gets to browsers more quickly.

Getting Superfish out of Firefox

First things first: If you are reading this post on a recent Lenovo laptop, please click the lock icon in the URL bar, then click “More Information…”.  If you see “Verified by: Superfish, Inc.”, you are infected with Superfish, and you should follow these instructions to remove it.

The Superfish adware distributed by Lenovo has brought the issue of SSL interception back to the headlines.  SSL interception is a technique that allows other software on a user’s computer to monitor and control their visits to secure Web sites — however, it also enables attackers to masquerade as secure websites, in order to spy on users or steal personal information.  Firefox is affected by Superfish, but Mozilla is deploying a hotfix to Firefox that works with other disinfection software to ensure that Firefox is disinfected as well.

Like other SSL interception software, Superfish seeks to add functionality to the Web by intercepting secure Web connections and injecting content into Web sites.  In order to be able to inject content into secure connections, it adds a trusted root certificate to the Windows and Firefox root stores.  With this trusted authority in place, Superfish can effectively create a fake ID for any website, so that it can convince Firefox that the browser is connected to the real website — even though it’s actually connected to Superfish.

This would be no worse than garden-variety adware if not for the fact that Superfish uses the same root certificate for all infected computers, and the private key for this certificate has been extracted and published to the Internet.  Using this private key, anyone on the Internet (not just Superfish) can create a fake ID that a Superfish-infected browser will accept.  So if you’re using a Superfish-infected computer to connect securely to your bank, you might actually be  connected to a criminal that is presenting a fake ID for your bank.

It appears that on affected systems (e.g., Lenovo laptops pre-loaded with Superfish), Superfish infects Firefox by adding its root certificate to the root store.  The good news is that according to research by Facebook and EFF, it appears that relatively few Firefox users have been infected.  The bad news is that some of the current disinfection tools do not disinfect Firefox.

For users that wish to ensure that they are disinfected, the best thing to do is to follow Lenovo’s instructions for removing Superfish.  This will remove Superfish entirely from the computer, including removing it from Firefox.

Some other disinfection tools will remove Superfish from Windows, but not from Firefox.  In order to ensure that these users are not vulnerable, we are deploying a hotfix today that detects whether Superfish has been removed, and if so, removes the Superfish root from Firefox.  We do not remove the root certificate if the Superfish software is still installed, since that would prevent the user from accessing any HTTPS websites.

Finally, a word to software authors who might be considering SSL interception: If you want to add features to the Web, don’t intercept, make an extension.  All of the major browsers offer extension frameworks (see these links for Firefox, Chrome, IE, Safari, and Opera).   Using these toolkits helps you avoid violating users’ security, while also giving you more powerful, and easier-to-use tools than you can get from an interception system.  The Web works better when we build it together.

MWoS – Audit-Go


The Mozilla Winter of Security of last year is concluding and the participating teams of students are completing their projects.

Our first team has completed the Audit-Go Heka plugin project recently with great success.
The Audit-Go plugin is a native Go implementation of a Linux Audit client. It communicates with the kernel using the Netlink protocol and has no extra dependencies.

The MWoS team and myself would like to thank our students Hardik Juneja, Arun Sori, Aalekh Nigam and their professor Sanjay Goel from the Jaypee Institute of Information Technology for their work and partnership during our mentoring sessions.

MWoS Audit-Go presentation on AirMozilla

The Mozilla Winter of Security logo is licensed under the Mozilla Public License, v. 2.0. You can obtain a copy of the license at

Phase 2: Phasing out Certificates with 1024-bit RSA Keys

In the previous post about certificates with 1024-bit RSA keys we said that the changes for the second phase of migrating off of 1024-bit root certificates were planned to be released in Firefox in early 2015. These changes have been made in Firefox 36, in which the following 1024-bit root certificates were either removed, or their SSL and Code Signing trust bits were turned off.

  • Verizon <>
    • CN = GTE CyberTrust Global Root
      • SHA1 Fingerprint: 97:81:79:50:D8:1C:96:70:CC:34:D8:09:CF:79:44:31:36:7E:F4:74
  • Symantec
    • CN = Thawte Server CA
      • SHA1 Fingerprint: 23:E5:94:94:51:95:F2:41:48:03:B4:D5:64:D2:A3:A3:F5:D8:8B:8C
    • CN = Thawte Premium Server CA
      • SHA1 Fingerprint: 62:7F:8D:78:27:65:63:99:D2:7D:7F:90:44:C9:FE:B3:F3:3E:FA:9A
    • OU = Class 3 Public Primary Certification Authority – G2
      • SHA1 Fingerprint: 85:37:1C:A6:E5:50:14:3D:CE:28:03:47:1B:DE:3A:09:E8:F8:77:0F
    • CN = Equifax Secure eBusiness CA-1
      • SHA1 Fingerprint: DA:40:18:8B:91:89:A3:ED:EE:AE:DA:97:FE:2F:9D:F5:B7:D1:8A:41

If you manage an SSL-enabled website, this change will not impact you if your certificates and the certificates above it have 2048-bit keys or more. If your SSL certificate has a 1024-bit key, or was issued by a certificate with a 1024-bit key, then you will need to get a new SSL certificate, and update the certificates in your Web server. If the intermediate certificate that you are using has a 1024-bit key, then you will need to download the 2048-bit intermediate certificate from the Certification Authority (CA), and update the certificate chain in your Web server. For your convenience, links to the impacted CAs are provided in the list above.

Please check your SSL certificates and replace any with 1024-bit RSA keys, and contact<> if you have comments or concerns.

Mozilla Security Engineering Team

Tighter Control Over Your Referrers

The purpose of the HTTP Referer (sic) header is to help sites figure out where their traffic comes from. However, as the Web got more complex, the amount of information in the Referer header ballooned, leading to bigger privacy problems. Firefox Beta supports a new feature to help sites protect their users’ privacy by changing the Referer header.

HTTP Referer provides a wealth of information about where you came from to the sites you visit, but this context isn’t always necessary (or desired). In addition, it is an unreliable tool for authenticating the origin of an HTTP request unless it’s always present, which it’s not due to privacy concerns (HTTPS sessions should not leak URLs to HTTP). When it is transmitted, there are still privacy concerns (“is that my username in the URL?”) because it’s all (whole URI) or nothing. To get what they want privacy-wise, sites often had to hack around direct loads with redirects and frames to change the referrer to something safer. What’s needed is a better way for referring sites to reduce the amount of data transmitted and thus providing a more uniform referrer that’s less privacy invasive.

This HTTP header has become quite problematic and not very useful, so we’re working to make it better.

Step one was to make gecko more flexible: we laid the groundwork so that it is easier for a user or browser extension to configure when Referer headers are sent and what they contain.

Step two is to help sites protect their users. Firefox 36 Beta now supports a feature called “meta referrer.” (Yes, this time “referrer” is spelled correctly.) Now your HTML documents can include a meta tag that specifies one of many referrer policies for the document to change what Firefox sends in the Referer header, and when it is sent. If your page contains the tag:

<meta name="referrer" content="origin">

all Referer headers in loads from your document will be without a path, query string or fragment, origin only. There are other policies you can specify to suppress referrers entirely, send a stripped-down referrer string cross-origin, and more.

We are proud to add this tool to the suite of features in Firefox that Web developers can use to protect their visitors’ privacy. Try it out! Let us know what you think!

Many thanks to Owen Chu for getting the gecko implementation started.

Sid Stamm
Security and Privacy Engineer

Mozilla at HITB Malaysia

The Mozilla security team was proud to be part of Hack In The Box (HITB) 2014, held from 15-16 October 2014 in Kuala Lumpur (KL), Malaysia.
IMG_5648Mozilla has been involved in HITB for several years now, and this year‘s HackWEEKDAY contest was probably the best we’ve seen so far. HackWEEKDAY is a contest where contestants develop mobile apps (Firefox OS or any other platforms allowed) in a bid to win glory and prizes. The competition was fierce this year, with over 75 developers and 4 hours of judging!

IMG_5750Notable entries included a Firefox extension which used Snort rules to block browser exploit attempts, a heat-activated mobile app to advise restaurant customers when the chef was cooking, and whole range of lock-screen apps –  from Morse code to facial recognition.

Mozilla also ran a booth promoting Firefox OS & WebMaker projects and the turnout was very encouraging. There was plenty of interest in Firefox OS phones, and thanks to the ground work from local community we had a constant stream of web maker experiments completed throughout the course of the conference (in exchange for T-shirts and goodies, of course!)
hitb_booth_demo   IMG_5902
Big thanks to the Mozilla Malaysia community Syafiq, Nasrun, Neo, Shah – among others for helping to make this a wonderful success, and being such gracious hosts (cendol durian, yum!) It’s sad that its the end an era for HITB KL, but looking forward to continue being a part of HITB Netherlands, and their new GSEC format events.
Paul and Gary, Mozilla Security Team

The POODLE Attack and the End of SSL 3.0


SSL version 3.0 is no longer secure. Browsers and websites need to turn off SSLv3 and use more modern security protocols as soon as possible, in order to avoid compromising users’ private information.

We have a plan to turn off SSLv3 in Firefox. This plan was developed with other browser vendors after a team at Google discovered a critical flaw in SSLv3, which can allow an attacker to extract secret information from inside of an encrypted transaction. SSLv3 is an old version of the security system that underlies secure Web transactions and is known as the “Secure Sockets Layer” (SSL) or “Transport Layer Security” (TLS).


In late September, a team at Google discovered a serious vulnerability in SSL 3.0 that can be exploited to steal certain confidential information, such as cookies. This vulnerability, known as “POODLE”, is similar to the BEAST attack. By exploiting this vulnerability, an attacker can gain access to things like passwords and cookies, enabling him to access a user’s private account data on a website.

Any website that supports SSLv3 is vulnerable to POODLE, even if it also supports more recent versions of TLS. In particular, these servers are subject to a downgrade attack, in which the attacker tricks the browser into connecting with SSLv3. This relies on a behavior of browsers called insecure fallback, where browsers attempt to negotiate lower versions of TLS or SSL when connections fail.

Today, Firefox uses SSLv3 for only about 0.3% of HTTPS connections. That’s a small percentage, but due to the size of the Web, it still amounts to millions of transactions per day.


The POODLE attack can be used against any browser or website that supports SSLv3. This affects all current browsers and most websites. As noted above, only 0.3% of transactions actually use SSLv3. Though almost all websites allow connections with SSLv3 to support old browsers, it is rarely used, since there are very few browsers that don’t support newer versions of TLS.

Sites that require SSLv3 will remain vulnerable until they upgrade to a more recent version of TLS. According to measurements conducted by Mozilla and the University of Michigan, approximately 0.42% of the Alexa top million domains have some reliance on SSLv3 (usually due to a subdomain requiring SSLv3).


SSLv3 will be disabled by default in Firefox 34, which will be released on Nov 25. The code to disable it is landing today in Nightly, and will be promoted to Aurora and Beta in the next few weeks. This timing is intended to allow website operators some time to upgrade any servers that still rely on SSLv3.

As an additional precaution, Firefox 35 will support a generic TLS downgrade protection mechanism known as SCSV. If this is supported by the server, it prevents attacks that rely on insecure fallback.

Additional Precautions

For Firefox users, the simplest way to stay safe is to ensure that Firefox is configured to automatically update. Look under Preferences / Advanced / Update and make sure that “Automatically install updates” is checked.

For users who don’t want to wait till November 25th (when SSLv3 is disabled by default in Firefox 34), we have created the SSL Version Control Firefox extension to disable SSLv3 immediately.

Website operators should evaluate their traffic now and disable SSLv3 as soon as compatibility with legacy clients is no longer required. (The only remaining browser that does not support TLSv1.0 is Internet Explorer 6). We recommend following the intermediate configuration level from Mozilla’s Server Site TLS guidelines.

We realize that many sites still receive traffic from IE6 and cannot disable SSLv3 entirely. Those sites may have to maintain SSLv3 compatibility, and should actively encourage their users to migrate to a more secure browser as soon as possible.

CSP for the web we have

Content Security Policy (CSP) is a good safety net against Cross Site Scripting (XSS). In fact, it’s the best one and I would recommend it to anyone building new sites.

For existing sites, implementing CSP can be a challenge because CSP introduces some restrictions by default and, if the code was written without these restrictions in mind, work will be required. Also, working around these issues can negate the benefits of applying a policy in the first place. In particular, inline scripts require thought; they’re commonly used and, if they’re allowed by your policy, the major benefit of CSP no longer applies. The only option available to make effective use of CSP, in the past, was to re-write the code to remove any existing inline scripts or styles.

Applying CSP to existing site might seem overwhelming at first but, considering the security benefit, the effort is well worth it. Fortunately, doing this has become much easier with CSP 2.

Some CSP 2 features:
CSP 2 provides some features that can really help; hash-source and nonce-source. These both provide a way of using inline scripts and styles without giving attackers free reign to inject things.

So how do they work? We’ll look at nonce-source first.

A CSP with a nonce-source might look like this:

content-security-policy: default-src 'self'; script-src 'nonce-2726c7f26c'

And the corresponding document might contain a script element that looks like this:

<script nonce="2726c7f26c">

There are 2 things to note here; firstly, it’s important that the nonce changes for each response (I’ve seen an example where it doesn’t!) and, secondly, it’s important that the nonce is sufficiently hard to predict.

Now, because the nonce changes in a way that isn’t predictable, the attacker doesn’t know what to inject and so, by only allowing script (or style) elements with valid nonce attributes, we can be sure that injections will fail.

And what about hash-source? Well, this is similar in that, again, a source in the CSP is used to ensure that a script or style element in the body is supposed to be there but the mechanism used differs. Rather than relying on an attribute on the script element in the document, hash-source provides a hash, in the CSP, of the script elements that are to be allowed in the document.

So for a script element like this:


You’d have a CSP containing a hash-source a bit like this:

content-security-policy: script-src 'sha256-cLuU6nVzrYJlo7rUa6TMmz3nylPFrPQrEUpOHllb5ic='

Obviously, you’d need to add a hash-source for each script or style you wanted to include in your document.

Which to use (and when):
So you may be wondering why there are two mechanisms (when both are designed to allow inline scripts and styles) and when you should use one rather than the other.
You should resist the temptation to use these mechanisms everywhere; these techniques are only intended for cases where removing inline scripts is not an option.

Nonce-source will be most useful in most cases because it is simpler. You only need to include a single source in your policy to cover a number of inline elements. The downside is that, since a nonce must only be used once, you need to generate a new header (and a new document) for each page load. This makes nonce-source a good option for dynamically generated pages, but completely unsuitable for static content.

Hash-source is more complicated; you have to generate hashes for each and every element you want to allow…. but, because it doesn’t rely on a value being unknown to an attacker, the CSP and the script element can remain the same. This makes hash-source a useful mechanism for protecting content that is served statically.

Words of warning:
Please be careful when using either of these mechanisms in dynamically generated content; if an attacker can inject content into something you’ve set a nonce attribute on (or something you generate a hash-source from) then you may have created a free bypass for an attacker.

The inline script restrictions imposed by CSP include script valued attributes (commonly used for DOM Level 0 event handlers, e.g. onclick); hash-source and nonce-source cannot help you with these.  Currently CSP does not provide mechanisms to apply directives to such script valued attributes but let’s see what the future brings!

RSA Signature Forgery in NSS


A flaw in the Network Security Services (NSS) library used by Firefox and other products allows attackers to create forged RSA certificates. Mozilla has released updates to fix this vulnerability and you should apply these updates to ensure your safety on the internet.

Impact to Users

Users on a compromised network could be directed to sites using a fraudulent certificate and mistake them for legitimate sites. This could deceive them into revealing personal information such as usernames and passwords. It may also deceive users into downloading malware if they believe it’s coming from a trusted site.


The following updates have been released for Mozilla client products:

Firefox 32.0.3
Firefox for Android 32.0.3
Firefox for Android 31.1.1
Firefox ESR 31.1.1
Firefox ESR 24.8.1

Thunderbird 31.1.2
Thunderbird 24.8.1

SeaMonkey 2.29.1

Updates are also available for Beta and other development versions of these products.

Most users will receive these as automatic updates. In addition, they are available from our website for those who have disabled automatic updates (or from the Play store in the case of Firefox for Android).

Other products which incorporate the NSS library should upgrade their copy of NSS to one of the following:

NSS 3.16.5
NSS 3.17.1


We would like to thank the following researchers for reporting this issue:

Antoine Delignat-Lavaud of Inria Paris in team Prosecco
The Advanced Threat Research team at Intel Security

Additional information can be found in our advisory.