{"id":5221,"date":"2011-09-08T09:56:00","date_gmt":"2011-09-08T16:56:00","guid":{"rendered":"http:\/\/blog.mozilla.org\/metrics\/?p=5221"},"modified":"2019-09-18T12:05:34","modified_gmt":"2019-09-18T19:05:34","slug":"understanding-dnt-adoption-within-firefox","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/metrics\/2011\/09\/08\/understanding-dnt-adoption-within-firefox\/","title":{"rendered":"Understanding DNT Adoption within Firefox"},"content":{"rendered":"<div><em><em>UPDATED 2011-09-08 11:55am PST: changed the description of how we store and retain IP address to be more accurate<\/em><\/em><\/p>\n<\/div>\n<div>On March 23rd, Mozilla launched its newest and most awesome browser: Firefox 4. Along with a plethora of features, including faster\u00a0performance, better security and the whole nine yards, Firefox 4 included a cutting edge privacy feature called Do No Track (DNT). For the uninitiated,\u00a0DNT simply tells sites &#8220;I don&#8217;t want to be tracked&#8221; via a HTTP header visible to all advertisers and publishers.<\/div>\n<div>Mozilla&#8217;s new <a title=\"Privacy Blog\" href=\"http:\/\/blog.mozilla.org\/privacy\">Privacy Blog<\/a>\u00a0 has several posts on the feature, including a new one today releasing a <a href=\"http:\/\/people.mozilla.org\/~tlowenthal\/resources\/mozilla_dnt-field-guide.pdf\"><span style=\"text-decoration: underline;\"><em>Do Not Track Field Guide <\/em>for developers<\/span><\/a>. Based on our current numbers, we&#8217;ve been seeing for several weeks now just under 5% of our users with DNT turned on within Firefox.<\/div>\n<div><a href=\"http:\/\/blog.mozilla.org\/metrics\/files\/2011\/09\/mozilla-dnt-user-adoption.png\"><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-large wp-image-5242\" title=\"mozilla dnt user adoption\" src=\"http:\/\/blog.mozilla.org\/metrics\/files\/2011\/09\/mozilla-dnt-user-adoption-1024x436.png\" alt=\"\" width=\"621\" height=\"264\" srcset=\"https:\/\/blog.mozilla.org\/metrics\/files\/2011\/09\/mozilla-dnt-user-adoption-1024x436.png 1024w, https:\/\/blog.mozilla.org\/metrics\/files\/2011\/09\/mozilla-dnt-user-adoption-300x127.png 300w, https:\/\/blog.mozilla.org\/metrics\/files\/2011\/09\/mozilla-dnt-user-adoption.png 1450w\" sizes=\"(max-width: 621px) 100vw, 621px\" \/><\/a><a href=\"http:\/\/blog.mozilla.org\/metrics\/files\/2011\/09\/DNT.png\"><br \/>\n<\/a><\/div>\n<div>The Mozilla team is all about experimenting. We love innovating new technologies that do good and benefit the community as a whole. As Firefox 4\u00a0kept breaking records, the peak was 5,500 downloads\/minute, (source: <a href=\"http:\/\/blog.mozilla.org\/blog\/2011\/03\/25\/the-first-48-hours-of-mozilla-firefox-4\/\">http:\/\/blog.mozilla.org\/blog\/2011\/03\/25\/the-first-48-hours-of-mozilla-firefox-4\/<\/a>)\u00a0we felt that it would be important to understand whether people were enabling DNT. Every Metrics guy lives and dies by the data. In late 2010, the metrics team gave a small talk on how we collect <a title=\"blog\" href=\"http:\/\/blog.mozilla.org\/data\/2010\/12\/30\/flume-hive-and-realtime-indexing-via-elasticsearch-2\/\">log data<\/a> (<a title=\"video ppt\" href=\"http:\/\/www.cloudera.com\/videos\/hw10_video_scalein_collecting_and_querying_log_data_in_near_real_time\">click here for the video ppt<\/a>). While that project has gone multiple\u00a0iterations over time, the basic premise is still the same:<\/p>\n<ul>\n<li>\n<div>Grab logs from multiple data-centers.<\/div>\n<\/li>\n<li>Split out anonymized and non-anonymized data into two separate files<\/li>\n<li>Store both sets of files in HDFS<\/li>\n<li>Create relevant partitions inside HIVE<\/li>\n<li>Query the data.<\/li>\n<li>Drool over the stats.<\/li>\n<\/ul>\n<p><strong><\/strong>(Non-anonymized data such as IP address has a 6-month retention policy and is deleted on expiration)<\/p>\n<\/div>\n<div>We decided to follow the same approach for calculating DNT stats. Once every day, each Firefox instance pings the AUS servers with respect to its\u00a0DNT status. The ping request looks something like this:<br \/>\n&#8220;DNT:-&#8221; User has <strong>NOT<\/strong> set DNT<br \/>\n&#8220;DNT:1&#8221; User <strong>HAS<\/strong> set DNT and does *<strong>not<\/strong>* wish to be tracked.<\/div>\n<div>\n<p><span style=\"color: #000000;\">Armed with the following data points, a simple HIVE query gives us DNT stats for a given day:<\/span><\/p>\n<p><em>\u00a0 \u00a0SELECT ds, dnt_type, \u00a0count(distinct ip_address) \u00a0FROM web_logs WHERE (request_url LIKE &#8216;%Firefox\/4.0%&#8217; OR request_url LIKE &#8216;%Firefox\/5.0%&#8217;\u00a0OR request_url LIKE &#8216;%Firefox\/6.0%&#8217;) AND dnt_type != &#8216;DNT:1, 1&#8217; AND ds = &#8216;$dateTime&#8217; GROUP BY ds, dnt_type ORDER BY ds desc;<\/em><\/p>\n<p>The above script is run on a nightly basis and the result is then plotted over a time graph, as included with this post.<\/p>\n<p><span style=\"color: #000000;\">One <strong>BIG<\/strong> caveat:<\/span><\/p>\n<p><span style=\"color: #000000;\">The DNT numbers are being undercounted, primarily because we use hashed IP address as proxy for counting a unique user. This means, while\u00a0there can be multiple users behind a given NAT with DNT set, the counter is incremented only once. This may account for why our numbers are a bit lower than those being reported by other groups, including the recent study of 100 million Firefox users conducted by<\/span> <a title=\"Krux Digital\" href=\"http:\/\/paidcontent.org\/article\/419-new-study-shows-use-of-do-not-track-is-on-the-rise\/\">Krux Digital<\/a><\/p>\n<p><span style=\"color: #000000;\">Possible Fix, <strong>NOT<\/strong>:<\/span><\/p>\n<p><span style=\"color: #000000;\">While it is possible to uniquely identify each instance of browser, doing so will require that we start tracking users, thereby defeating the exact purpose for why\u00a0DNT was created in the first place.<\/span><\/p>\n<p>&nbsp;<\/p>\n<\/div>\n<div><span style=\"color: #000000;\">Feel free to leave us a comment or email: (aphadke at_the_rate mozilla dot com &#8211; Anurag Phadke) for more information.<\/span><\/div>\n","protected":false},"excerpt":{"rendered":"<p>UPDATED 2011-09-08 11:55am PST: changed the description of how we store and retain IP address to be more accurate On March 23rd, Mozilla launched its newest and most awesome browser: Firefox 4. Along with a plethora of features, including faster\u00a0performance, &hellip; <a class=\"go\" href=\"https:\/\/blog.mozilla.org\/metrics\/2011\/09\/08\/understanding-dnt-adoption-within-firefox\/\">Continue reading<\/a><\/p>\n","protected":false},"author":1557,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3647],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/metrics\/wp-json\/wp\/v2\/posts\/5221"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/metrics\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/metrics\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/metrics\/wp-json\/wp\/v2\/users\/1557"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/metrics\/wp-json\/wp\/v2\/comments?post=5221"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/metrics\/wp-json\/wp\/v2\/posts\/5221\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/metrics\/wp-json\/wp\/v2\/media?parent=5221"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/metrics\/wp-json\/wp\/v2\/categories?post=5221"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/metrics\/wp-json\/wp\/v2\/tags?post=5221"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}