Following the launch of Tiles in November, I wanted to provide more information on how data is transmitted into and from Firefox. Last week, I described how we get Tiles data into Firefox differently from the usual cookie-identified requests. In this post, I will describe how we report on users’ interactions with Tiles.
As a reminder, we have three kinds of Tiles: the History Tiles, which were implemented in Firefox in 2012, Enhanced Tiles, where we have a custom creative design for a Tile for a site that a user has an existing relationship with, and Directory Tiles, where we place a Tile in a new tab page for users with no browsing history in their profile. Enhanced and Directory Tiles may both be sponsored, involving a commercial relationship, or they may be Mozilla projects or causes, such as our Webmaker initiative.
We need to be able to report data on user’s interactions with Tiles for two main reasons:
- to determine if the experience is a good one
- to report to our commercial partners on volumes of interactions by Firefox users
And we do these things in accordance with our data principles both to set the standards we would like the industry to follow and, crucially, to maintain the trust of our users.
Unless a user has opted out by switching to Classic or Blank, Firefox currently sends a list of the Tiles on a user’s new tab page to Mozilla’s servers, along with data about the user’s interaction with the Tiles, e.g., view, click, or pin.
Directory and Enhanced Tiles are identified by a Tile id, (e.g., “Firefox for Android” Tile has an id of 499 for American English-speaking users while “Firefox pour Android” has an id of 510 for French-speaking users). History Tiles do not have an id, so we can only know that the user saw a history screenshot but not what page — except for early release channel Telemetry related experiments, we do not currently send URL information for Tiles, although of course we are able to infer it for the Directory and Enhanced Tiles that we have sent to Firefox.
Our implementation of Tiles uses the minimal actionable dataset, and we protect that data with with multiple layers of security. This means:
- cookie-less requests
- encrypted transmission
- aggressive cleaning of data
We also break up the data into smaller pieces that cannot be reconstructed to the original data. When our server receives a list of seen Tiles from an IP address, we record that the specific individual Tiles were seen and not the whole list.
With the data aggregated across many users, we can now calculate how many total times a given Tile has been seen and visited. By dividing the number of clicks by the number of views, we get a click-through-rate (CTR) that represents how valuable users find a particular tile, as well as a pin-rate and a block-rate. This is sufficient for us to determine both if we think a Tile is useful for a user and also for us to report to a commercial partner.
Calculating the CTR for each tile and comparing them helps us decide if a Tile is useful to many users. We can already see that the most popular tiles are “Customize Firefox” and “Firefox for Android” (Tile 499, remember) both in terms of clicks and pins.
For an advertiser, we create reports from our aggregated data, and they in turn can see the traffic for their URLs and are able to measure goal conversions on their back end. Since the Firefox 10th anniversary announcement, which included Tiles and the Firefox Developer Edition, we ran a Directory Tile for the Webmaker initiative. After 25 days, it had generated nearly 1 billion views, 183 thousand clicks, and 14 thousand pins.
The Webmaker team, meanwhile, are able to see the traffic coming in (as the Tile directs traffic to a distinct URL), and they are able to give attribution to the Tile and track conversions from there:
We started with a relatively straightforward implementation to be able to measure how users are interacting with Tiles. But we’ve already gotten some good ideas on how to make things even better for improved accuracy with less data. For example, we currently cannot accurately measure how many unique users have seen a given Tile, and traditionally unique identifiers are used to measure that, but HyperLogLog has been suggested as a privacy-protecting technique to get us that data. A separate idea is that we can use statistical random sampling that doesn’t require all Firefox users to send data while still getting the numbers we need. We’ll test sampling through Telemetry experiments to measure site popularity, and we’ll share more when we get those results.
We would love to hear your thoughts on how we treat users data to find the Tiles that users want. And if you have ideas on how we can improve our data collection, please send them over as well!
– Ed Lee on behalf of the Tiles team.