Improving privacy without breaking the web

First: thank you to our passionate and active Firefox users who participated in this shield study!

tl;dr – The Firefox Privacy team ran a user research study to learn how privacy protections affect users on websites. We learned some surprising things. There were 19,000 users and 8 variations of behavior within the experiment. We built an opt-in study to measure breakage data, we unblocked some existing privacy features, and we learned some new potential areas to improve privacy in the future. And as a result, we’re adding more privacy protection to Firefox:

  1. In Firefox Quantum, all users can enable Tracking Protection for their regular browsing
  2. In Firefox 59+, Private Browsing will default to trimming Referer values to origins

(Note: You can also see the full presentation of these results)

Existing Knowledge, Assumptions, and Questions

For over a decade, Mozilla has been building privacy protections for Internet users. From Firefox desktop, to Firefox mobile and specialty browsers, to private encrypted web services like Send, we continuously strive to learn how to improve privacy technology across the web.

Recently, the Firefox Telemetry and Data platform helped us answer some long-standing questions for Firefox desktop privacy:

  • Does Tracking Protection break websites?
  • Do broken websites make users leave Firefox?
  • Are there existing privacy protections we could enable with minimal web breakage?

The shield study add-on

To help answer these questions, we built an opt-in shield study. We placed each user into one of nine branches of the study. Each branch corresponded to an existing Firefox privacy protection.

  • Control
    No changes
  • sessionOnlyThirdPartyCookies
    When the user closes Firefox, Firefox deletes third-party cookies.
  • noThirdPartyCookies
    Firefox disables all third-party cookies.
  • thirdPartyCookiesOnlyFromVisitedFirefox does not send third-party cookies to a site unless the user directly visited the site in the past.
  • trackingProtection
    Activates tracking protection in regular browsing windows.
  • originOnlyRefererToThirdParties
    Trim requests’ Referer values to origins when sent to third parties.
  • resistFingerprinting
    Activates Firefox’s fingerprinting protections.
  • firstPartyIsolation
    Activates First-party Isolation.
  • firstPartyIsolationOpenerAccess
    Activates First-party Isolation, but allows pages to access openers.

Once a user was placed into a branch, we gave them a new browser toolbar icon to report problems. See the full presentation for a screenshot flow of the add-on experience.

The numbers

Over 19,000 users opted into the study, which gave us more than 2,100 users in each branch of the study, and over 8,500 active users on the most active day of the study.

Pie chart of users in each branch

2,100+ users in each branch

 

Chart of active users per day

Up to 8,500 active users per day

Measuring breakage

To quantify web breakage, we analyzed the data by 3 primary dimensions:

  • % of users who reported at least one problem
  • Average number of problems reported per user
  • % of users who disable the study (presumably because of problems)

We also analyzed the types of breakage, and those details are available in the full presentation of the results of the study.

Tracking Protection actually reduces problems

Firefox has had Tracking Protection built into its Private Browsing Mode since 2015. Tracking Protection blocks all third-party connections to domains on Disconnect‘s Tracking Protection block-list. We know that this breaks some websites where the code relies on the third-party resources. (We have a bug tree and a long list of webcompat.com issues for the Firefox feature, and we ran a Test Pilot experiment with the same block-list.)

In this study, we measured and compared breakage caused by Tracking Protection to a control group, and to breakage caused by other protections. Which led to our first surprising result …

Chart of average problems per user

The average problems reported per user of Tracking Protection was lower than the control group.

 

When we saw this, we dug into users’ comments to learn why. We saw a trend among the comments from users in the control group: “not responsive”, “slow”, “freezing”, “took longer to load”, “not always responding”, “laggy”, “doesn’t load fast” … and the comment that seemed to sum it all up:

Something on the page is slowing down the loading speed significantly.

Our finding here matches what web performance guru’s have been saying for years: third-party scripts cause a large number of performance problems. Tracking Protection removes them completely, so the number of problems is reduced. So, in a sense, Tracking Protection may actually fix websites by blocking tracking elements that break (i.e., slow) them down.

Do broken websites make users leave Firefox?

Privacy & Security engineers have long understood: “without usable systems, the security and privacy simply disappears“. Firefox’s privacy protections must be usable on the web, or people will simply stop using Firefox altogether. While we could not measure the number of users who stopped using Firefox, we did measure the number of users who disabled the study.

Unsurprisingly, some privacy protections caused significantly more users to disable the study than others.

Chart of % of users who disabled the study

Significantly more users disabled resistFingerprinting and firstPartyIsolation branches of the study.

 

Surprisingly, though, the % of users disabling the study was low across all branches: between 5.7% minimum and 9.7% maximum. Furthermore, the % of users who disabled Tracking Protection, Origin-only Referer values to third parties, and any of the cookie protections were within the margin-of-error of the control group. This result indicates that, overall, many privacy protections don’t appear to break the web so much that users will disable them.

However, we did analyze the kinds of breakage that users reported, and we learned some specific broken websites and specific broken features that correlated to more users disabling the study. The details are available in the full presentation. In short, breaking “workflow” sites and features caused more people to disable the study.

Are there existing privacy protections we could enable with minimal web breakage?

To learn which branches of privacy protection were associated with the least overall breakage, we looked at each of our three dimensions to see which protections fell within a margin of error of the control group.

% of users reporting at least 1 problem: 6 protections are within the margin of error of the control group

% of users reporting at least 1 problem; 6 protections are within the margin of error of the control group

 

average problems per user: 4 protections are within the margin of error of the control group

Average problems reported per user; 4 protections are within the margin of error of the control group

 

% of users who disabled the study: 5 protections are within the margin of error of the control group

% of users who disabled the study; 5 protections are within the margin of error of the control group

 

We created a simple “composite breakage score” that multiplied these three dimensions together for a consolidated comparison. The graph below is a view of the data that emphasizes the relative differences.

"Composite Breakage Score" for each privacy protection

By this comparison, the most promising protections, in terms of lowest overall breakage were:

  1. Origin-only Referer values to third parties
  2. Session-only third-party cookies
  3. Tracking Protection

Data turns into action

After this study concluded, we presented the results to a number of teams, and we’re happy that a couple of strong decisions and actions are already made and underway.

  1. In Firefox Quantum, all users can enable Tracking Protection for their regular browsing
  2. In Firefox 59+, Private Browsing will default to trimming Referer values to origins

In conclusion, we built an opt-in study to measure breakage data, we unblocked some existing privacy features, and we learned some new potential areas to improve privacy in the future. We look forward to using more data to improve privacy on the web.

Tags:

Categories: Uncategorized