Effectively Measuring Search in Firefox

Last year, we announced new telemetry measurements to count follow-on searches that start in Firefox and continue on search engine pages. This follow-on measurement helped us understand our users and our business, but we’ve come to realize we still have major gaps.

To address this challenge, we are deploying additional search measurements that will give us further insight into how users search with our browser. The new measurements will give a fuller picture of search activity while maintaining user privacy by limiting collection to aggregate counts. We want to understand aggregate behavior and potential for revenue – basically how users are searching and how often they are searching – but not what they are searching for.

Mozilla doesn’t vacuum up your data and worry about the consequences later. While we compete in an industry that is driven by data, we strictly follow a set of data privacy principles that limit what we learn about our own products.

Mozilla continues to stand behind that approach. But there are business and competitive consequences to our approach –  we don’t always know as much as we want about the fundamentals of our own business. This is a healthy problem to have; it shows we’ve erred on the side of respecting and protecting our users. But it also presents a challenge for us and it puts us at a disadvantage when even our own search partners have more expansive data than we do about our own business. Search is how Mozilla generates its revenue, so that disadvantage has real impact on our ability to fund work to build a healthier internet.

Here is an overview of the changes we are making to measure search more effectively. The first – organic search measurement – launched in Firefox two weeks ago. The two others, focused on measuring ad clicks and telemetry opt-out rates – will launch in the near future.

Organic Search

Last year we launched the in-content search probe, which counts the number of queries Firefox users issue to our search partners. Previously, this exclusively counted queries when users started their search using one of our built-in search tools, like the awesome bar. These are the searches that give Mozilla revenue. We’ve expanded this telemetry to include a count of the number of times a user directly navigates to a search provider. This will simplify our data collection tooling, and give us insight into all the ways users are searching within Firefox.

We will continue to only collect a count of queries. We do not collect information on the content of your search queries (the words you type into the search box) nor other web browsing data.

Ad Click Measurement

We also plan to count the number of times a search page displays ads and the number of times users click ads. These will be counts by user. Mozilla will not know the content of the search nor the content of the ads. This helps us both forecast Mozilla revenue and also understand the impact of ad blocking on the larger web ecosystem.

Telemetry Coverage

Finally, we need better insight into our opt-out rates for telemetry. We use telemetry to ensure new features improve your user experience and to guide Mozilla’s business decisions. However, an unknown portion of our users do not report telemetry for a variety of reasons. This means we may not have data that is representative of our entire population. For example, some enterprise builds are preconfigured to not send telemetry and some users manually opt-out of telemetry collection. We believe the large majority of clients do send telemetry but currently have no way of measuring this.

To address this, we will measure Telemetry Coverage, which is the percentage of all Firefox users who report telemetry. The Telemetry Coverage measurement will sample a portion of all Firefox clients and report whether telemetry is enabled. This measurement will not include a client identifier and will not be associated with our standard telemetry.

As always, you’ll be able to find the full details about these measurements in public documentation for all telemetry collected within Firefox. And this data collection will go through our data review process to ensure we these approaches are appropriately vetted by our Data Stewards.

We want to deliver a product that meets the needs and expectations of our users. We also want to make sure we can compete in a market where other companies treat data as a commodity.  We don’t want or need all of the data that others collect, but data can help us deliver a better, faster product for our users while respecting their privacy, security, and choices.

Marshall Erwin, Director of Trust & Security