Improving privacy without breaking the web

First: thank you to our passionate and active Firefox users who participated in this shield study!

tl;dr – The Firefox Privacy team ran a user research study to learn how privacy protections affect users on websites. We learned some surprising things. There were 19,000 users and 8 variations of behavior within the experiment. We built an opt-in study to measure breakage data, we unblocked some existing privacy features, and we learned some new potential areas to improve privacy in the future. And as a result, we’re adding more privacy protection to Firefox:

1. In Firefox Quantum, all users can enable Tracking Protection for their regular browsing
2. In Firefox 59+, Private Browsing will default to trimming Referer values to origins

(Note: You can also see the full presentation of these results)

Existing Knowledge, Assumptions, and Questions

For over a decade, Mozilla has been building privacy protections for Internet users. From Firefox desktop, to Firefox mobile and specialty browsers, to private encrypted web services like Send, we continuously strive to learn how to improve privacy technology across the web.

Recently, the Firefox Telemetry and Data platform helped us answer some long-standing questions for Firefox desktop privacy:

• Does Tracking Protection break websites?
• Do broken websites make users leave Firefox?
• Are there existing privacy protections we could enable with minimal web breakage?

The shield study add-on

To help answer these questions, we built an opt-in shield study. We placed each user into one of nine branches of the study. Each branch corresponded to an existing Firefox privacy protection.

• Control
  No changes
• sessionOnlyThirdPartyCookies
  When the user closes Firefox, Firefox deletes third-party cookies.
• noThirdPartyCookies
  Firefox disables all third-party cookies.
• thirdPartyCookiesOnlyFromVisitedFirefox does not send third-party cookies to a site unless the user directly visited the site in the past.
• trackingProtection
  Activates tracking protection in regular browsing windows.
• originOnlyRefererToThirdParties
  Trim requests’ Referer values to origins when sent to third parties.
• resistFingerprinting
  Activates Firefox’s fingerprinting protections.
• firstPartyIsolation
  Activates First-party Isolation.
• firstPartyIsolationOpenerAccess
  Activates First-party Isolation, but allows pages to access openers.

Once a user was placed into a branch, we gave them a new browser toolbar icon to report problems. See the full presentation for a screenshot flow of the add-on experience.

The numbers

Over 19,000 users opted into the study, which gave us more than 2,100 users in each branch of the study, and over 8,500 active users on the most active day of the study.

 

Measuring breakage

To quantify web breakage, we analyzed the data by 3 primary dimensions:

• % of users who reported at least one problem
• Average number of problems reported per user
• % of users who disable the study (presumably because of problems)

We also analyzed the types of breakage, and those details are available in the full presentation of the results of the study.

Tracking Protection actually reduces problems

Firefox has had Tracking Protection built into its Private Browsing Mode since 2015. Tracking Protection blocks all third-party connections to domains on Disconnect‘s Tracking Protection block-list. We know that this breaks some websites where the code relies on the third-party resources. (We have a bug tree and a long list of webcompat.com issues for the Firefox feature, and we ran a Test Pilot experiment with the same block-list.)

In this study, we measured and compared breakage caused by Tracking Protection to a control group, and to breakage caused by other protections. Which led to our first surprising result …

 

When we saw this, we dug into users’ comments to learn why. We saw a trend among the comments from users in the control group: “not responsive”, “slow”, “freezing”, “took longer to load”, “not always responding”, “laggy”, “doesn’t load fast” … and the comment that seemed to sum it all up:

Something on the page is slowing down the loading speed significantly.

Our finding here matches what web performance guru’s have been saying for years: third-party scripts cause a large number of performance problems. Tracking Protection removes them completely, so the number of problems is reduced. So, in a sense, Tracking Protection may actually fix websites by blocking tracking elements that break (i.e., slow) them down.

Do broken websites make users leave Firefox?

Privacy & Security engineers have long understood: “without usable systems, the security and privacy simply disappears“. Firefox’s privacy protections must be usable on the web, or people will simply stop using Firefox altogether. While we could not measure the number of users who stopped using Firefox, we did measure the number of users who disabled the study.

Unsurprisingly, some privacy protections caused significantly more users to disable the study than others.

 

Surprisingly, though, the % of users disabling the study was low across all branches: between 5.7% minimum and 9.7% maximum. Furthermore, the % of users who disabled Tracking Protection, Origin-only Referer values to third parties, and any of the cookie protections were within the margin-of-error of the control group. This result indicates that, overall, many privacy protections don’t appear to break the web so much that users will disable them.

However, we did analyze the kinds of breakage that users reported, and we learned some specific broken websites and specific broken features that correlated to more users disabling the study. The details are available in the full presentation. In short, breaking “workflow” sites and features caused more people to disable the study.

Are there existing privacy protections we could enable with minimal web breakage?

To learn which branches of privacy protection were associated with the least overall breakage, we looked at each of our three dimensions to see which protections fell within a margin of error of the control group.

 

 

 

We created a simple “composite breakage score” that multiplied these three dimensions together for a consolidated comparison. The graph below is a view of the data that emphasizes the relative differences.

By this comparison, the most promising protections, in terms of lowest overall breakage were:

1. Origin-only Referer values to third parties
2. Session-only third-party cookies
3. Tracking Protection

Data turns into action

After this study concluded, we presented the results to a number of teams, and we’re happy that a couple of strong decisions and actions are already made and underway.

1. In Firefox Quantum, all users can enable Tracking Protection for their regular browsing
2. In Firefox 59+, Private Browsing will default to trimming Referer values to origins

In conclusion, we built an opt-in study to measure breakage data, we unblocked some existing privacy features, and we learned some new potential areas to improve privacy in the future. We look forward to using more data to improve privacy on the web.