(Note: You can also see the full presentation of these results<\/a>)<\/p>\n For over a decade, Mozilla has been building privacy protections for Internet users. From Firefox desktop<\/a>, to Firefox mobile<\/a> and specialty<\/a> browsers<\/a>, to private encrypted web services like Send,<\/a> we continuously strive to learn how to improve privacy technology across the web.<\/p>\n Recently, the Firefox Telemetry and Data platform helped us answer some long-standing questions for Firefox desktop privacy<\/b>:<\/p>\n To help answer these questions, we built an opt-in shield study<\/a>. We placed each user into one of nine branches of the study. Each branch corresponded to an existing Firefox privacy protection<\/b>.<\/p>\n Once a user was placed into a branch, we gave them a new browser toolbar icon to report problems. See the full presentation for a screenshot flow of the add-on experience<\/a>.<\/p>\n Over 19,000 users opted into the study, which gave us more than 2,100 users in each branch of the study, and over 8,500 active users on the most active day of the study.<\/p>\n 2,100+ users in each branch<\/p><\/div>\n <\/p>\n Up to 8,500 active users per day<\/p><\/div>\n To quantify web breakage, we analyzed the data by 3 primary dimensions<\/strong>:<\/p>\n We also analyzed the types of breakage, and those details are available in the full presentation of the results of the study<\/a>.<\/p>\n Firefox has had Tracking Protection built into its Private Browsing Mode since 2015. Tracking Protection blocks all third-party connections to domains on Disconnect<\/a>‘s Tracking Protection block-list. We know that this breaks some websites where the code relies on the third-party resources. (We have a bug tree<\/a> and a long list of webcompat.com<\/a> issues for the Firefox feature, and we ran a Test Pilot experiment<\/a> with the same block-list.)<\/p>\n In this study, we measured and compared breakage caused by Tracking Protection to a control group<\/strong>, and to breakage caused by other protections. Which led to our first surprising result …<\/p>\n The average problems reported per user of Tracking Protection was lower than the control group.<\/p><\/div>\n <\/p>\n When we saw this, we dug into users’ comments to learn why. We saw a trend among the comments from users in the control<\/strong> group: “not responsive”, “slow”, “freezing”, “took longer to load”, “not always responding”, “laggy”, “doesn’t load fast” … and the comment that seemed to sum it all up:<\/p>\n Something on the page is slowing down the loading speed significantly.<\/p><\/blockquote>\n Our finding here matches what web performance guru’s have been saying for years<\/a>: third-party scripts cause a large number of performance problems. Tracking Protection removes them completely, so the number of problems is reduced. So, in a sense, Tracking Protection may actually fix<\/span> websites by blocking tracking elements that break (i.e., slow) them down<\/strong>.<\/p>\n Privacy & Security engineers have long understood: “without usable systems, the security and privacy simply disappears<\/a><\/strong>“. Firefox’s privacy protections must be usable on the web, or people will simply stop using Firefox altogether. While we could not measure the number of users who stopped using Firefox, we did measure the number of users who disabled the study<\/strong>.<\/p>\n Unsurprisingly, some privacy protections caused significantly more users to disable the study than others.<\/p>\n Significantly more users disabled resistFingerprinting and firstPartyIsolation branches of the study.<\/p><\/div>\n <\/p>\n Surprisingly, though, the % of users disabling the study was low across all branches: between 5.7% minimum and 9.7% maximum. Furthermore, the % of users who disabled Tracking Protection, Origin-only Referer values to third parties, and any of the cookie protections were within the margin-of-error of the control group. This result indicates that, overall, many privacy protections don’t appear to break the web so much that users will disable them<\/strong>.<\/p>\n However, we did analyze the kinds of breakage that users reported, and we learned some specific broken websites and specific broken features that correlated to more users disabling the study. The details are available in the full presentation<\/a>. In short, breaking “workflow” sites and features caused more people to disable the study<\/strong>.<\/p>\n To learn which branches of privacy protection were associated with the least overall breakage, we looked at each of our three dimensions to see which protections fell within a margin of error of the control group.<\/p>\n % of users reporting at least 1 problem; 6 protections are within the margin of error of the control group<\/p><\/div>\n <\/p>\n Average problems reported per user; 4 protections are within the margin of error of the control group<\/p><\/div>\n <\/p>\n % of users who disabled the study; 5 protections are within the margin of error of the control group<\/p><\/div>\n <\/p>\n We created a simple “composite breakage score” that multiplied these three dimensions together for a consolidated comparison. The graph below is a view of the data that emphasizes the relative differences.<\/p>\n By this comparison, the most promising protections, in terms of lowest overall breakage were:<\/p>\n After this study concluded, we presented the results to a number of teams, and we’re happy that a couple of strong decisions and actions are already made and underway.<\/p>\n In conclusion, we built an opt-in study to measure breakage data, we unblocked some existing privacy features, and we learned some new potential areas to improve privacy in the future. We look forward to using more data to improve privacy on the web.<\/p>\n","protected":false},"excerpt":{"rendered":" First: thank you to our passionate and active Firefox users who participated in this shield study! tl;dr \u2013 The Firefox Privacy team ran a user research study to learn how … Read more<\/a><\/p>\n","protected":false},"author":285,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[323282],"tags":[847],"coauthors":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/posts\/55"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/users\/285"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/comments?post=55"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/posts\/55\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/media?parent=55"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/categories?post=55"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/tags?post=55"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/coauthors?post=55"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Existing Knowledge, Assumptions, and Questions<\/h2>\n
\n
The shield study add-on<\/h2>\n
\n
\n<\/strong>No changes<\/li>\n
\n<\/strong>When the user closes Firefox, Firefox deletes third-party cookies<\/a>.<\/li>\n
\n<\/strong>Firefox disables all third-party cookies<\/a>.<\/li>\n
\n<\/strong>Activates tracking protection<\/a> in regular browsing windows.<\/li>\n
\n<\/strong>Trim requests’ Referer<\/code> values to origins when sent to third parties.<\/li>\n
\n<\/strong>Activates Firefox’s fingerprinting<\/a> protections.<\/li>\n
\n<\/strong>Activates First-party Isolation.<\/li>\n
\n<\/strong>Activates First-party Isolation, but allows pages to access openers.<\/li>\n<\/ul>\n<\/h2>\n
The numbers<\/h2>\n
![\"Pie](\"http:\/\/blog.mozilla.org\/data\/files\/2018\/01\/priv-prefs-shield-study-branches-580x207.png\")
<\/a>![\"Chart](\"http:\/\/blog.mozilla.org\/data\/files\/2018\/01\/priv-prefs-shield-study-active-users-580x157.png\")
<\/a><\/h2>\n
Measuring breakage<\/h2>\n
\n
<\/h2>\n
Tracking Protection actually reduces problems<\/h2>\n
![\"Chart](\"http:\/\/blog.mozilla.org\/data\/files\/2018\/01\/priv-prefs-shield-study-avg-prob-580x143.png\")
<\/a><\/h2>\n
Do broken websites make users leave Firefox?<\/h2>\n
![\"Chart](\"http:\/\/blog.mozilla.org\/data\/files\/2018\/01\/priv-prefs-shield-study-disables-580x143.png\")
<\/a><\/h2>\n
Are there existing privacy protections we could enable with minimal web breakage?<\/h2>\n
![\"%](\"http:\/\/blog.mozilla.org\/data\/files\/2018\/01\/priv-prefs-shield-study-percent-users-margin-580x164.png\")
<\/a>![\"average](\"http:\/\/blog.mozilla.org\/data\/files\/2018\/01\/priv-prefs-shield-study-avg-problems-margin-580x181.png\")
<\/a>![\"%](\"http:\/\/blog.mozilla.org\/data\/files\/2018\/01\/priv-prefs-shield-study-disables-margin-580x168.png\")
<\/a>![\""Composite](\"http:\/\/blog.mozilla.org\/data\/files\/2018\/01\/priv-prefs-shield-study-composite-breakage-scores-580x239.png\")
<\/a><\/p>\n\n
<\/h2>\n
Data turns into action<\/h2>\n
\n