Mozilla Open Data Visualization Contest – Data is Now Available!

[Note: cross-posted on Mozilla Labs]

Two weeks ago the Mozilla Metrics Team, together with Mozilla Labs and the growing Mozilla Research Initiative, announced our first Open Data Visualization Competition. Today, we are excited to release the data sets for this competition!

These data sets come from Mozilla’s own open data program, Test Pilot. Test Pilot is a user research platform that collects structured user data through Firefox. Currently, over 1 million Firefox users from all over the world participate in Test Pilot studies, which aim to explore how people use their web browser and the Internet in general.

For this challenge, data samples from two recent Test Pilot studies have been made available – check out the data sample pages below for a thorough description:

In addition, please note that the submission deadline has been extended from Dec. 5 to Dec. 17th to encourage more participation. Be sure to visit the official Competition page for more general info on dates, judges, and prizes.

Good luck and start hacking!

Bouncing Off of Menus

One of the most significant changes to the user interface for Firefox 4 was the rolling of the old menu items into one button, something we have called on this blog the “combined menu.” We’ve addressed the impact of this change in other posts, such as this one.  We thought of one obvious follow-up: how often do people ‘bounce away’ from the combined menu before clicking something in it?

We define the bounce rate as this: given a user clicking the combined menu button, what is the proportion of instances where users do not click on one of the menu items? We built a small NumPy library to help us analyze the sequence of user-browser interactions.

44% of users clicked something other than a menu item after having clicked the Firefox button, while 56% clicked on a succeeding menu item. These percentages do not change even if we throw out the first few instances of every user clicking on the Firefox button (new Beta users are likely curious about the new menu layout, so try it out).

The bounce rate for this study has gone up since the last version (where about 38% of button clicks led to bounces).  One possible explanation points to the handful new items added to the combined menu since the last beta interface study.  The additional cognitive load might lead to more bounces.

We are currently working on calculating the bounce rate for the pre-combined menu UI.  Stay tuned for that.

Mozilla Open Data Visualization Competition – How Do People Use Firefox

The Mozilla Metrics Team, together with Mozilla Labs and the growing Mozilla Research Initiative, is excited to announce our first Open Data Visualization Competition.

Using data from Mozilla’s own open data program, Test Pilot, we’d like to explore creative visual answers to the question: “How do people use Firefox?” We are looking for compelling visualizations that tell detailed, meaningful and yet easy-to-interpret stories about interesting user activities.

Read on for details on the data (accessible Nov. 17), prizes, and judges, including special judge, David Smith from Revolution Analytics. Also, don’t forget to check out the official Competition page and follow @moztestpilot on Twitter for news and updates.

The Data

This competition is based on Mozilla’s own open data program, Test Pilot. Test Pliot is a user research platform that collecting structured user data through Firefox. All data is gathered through pre-defined Test Pilot studies which aim to explore how people use their web browser and the Internet.

Currently, over 1 million Firefox users from all over the world participate in Test Pilot studies. The goal for this platform is to encourage everyone from all skill levels to improve the Web experience by conducting and participating in these studies. Test Pilot study results are made available under open licenses, with the data being anonymized before release. (For more information about the Test Pilot data policy, please check Privacy Policy.)

For this challenge, we will use data from two recent Test Pilot studies:

Partners and Judges

We are honored to have David Smith from Revolution Analytics partnering with us and serving as a special judge. Members from Mozilla Labs and Mozilla Metrics will form the rest of the judging panel.


To recognize the awesome work from participants, some great prizes are at stake:

  • Grand prize will be a $300 Amazon gift card
  • Two “Best in Class” teams will receive a set of Edward Tufte’s books

We’ll also present all submissions on the Test Pilot website and Mozilla Metrics blog in a special post to highlight your work.

Join the competition!

You can choose any tools you like for your analysis and visualization, including but not limited to: R, Matlab, Protovis, Processing or IBM many eyes. You can participate solo or team up with other people. You are also welcome to enter as many times as you like. If you are interested to join the competition, please follow the following important dates:

  • Nov. 17th: Check the official Competition page on Nov. 17th to download the data
  • Dec.5th : Go here to submit your results and enter the competition before Dec.5th.
  • Dec. 14th: Winning visualizations will be announced on Dec. 14th.

To facilitate the free exchange of ideas, all visualizations and other contributions you make to this challenge must be contributed under the Creative Commons Attribution-ShareAlike 3.0 license.

About the New ‘About Firefox’ Window

Earlier this year, Test Pilot ran a Menu Item Usage Study to better understand how users interact with the traditional menu bar as Firefox transitions to a more streamlined, single “application button.” We’ve already shared some analysis from this study: we answered which menu items are the most and least commonly used, explored the distribution of menu interactions, and learned that most users actually use very few commands from the menu bar.

We also examined the sequence of menu bar actions – specifically looking for common sequences in commands. The strongest connection was between the ‘About Firefox’ and ‘Check for Updates’ menu items; after clicking ‘About Firefox,’ 9% of users click ‘Check for Updates’ within 20 seconds.

From this data, we reasonably assume that many users go to “About Firefox” to check whether they should update. Accordingly, three minor changes to these menu items should improve user experience.

  1. Move ‘Check for Updates’ from the ‘Help’ header to the ‘Firefox’ header directly under ‘About Firefox’ so these related commands are logically grouped together (on Mac OSX).
  2. Add a ‘Check for Updates’ button directly to the ‘About Firefox’ window.
  3. If automatic updates are set, initialize a check for updates when ‘About Firefox’ opens, and give the user the option to update directly from the window (if any updates are available).

Luckily, we have some excellent engineers working on Firefox who came to the same conclusions.  Margaret Leibovic, one of these awesome engineers, recently blogged about her first month’s work here at Mozilla –  this work included implementing the first 2 of these changes.  Check out the new ‘About Firefox’ window below:

The relocation of ‘Check for Updates’ and this refreshed window have already landed in Minefield, and will be integrated into Firefox 4 from beta 7. Robert Strong, another Firefox Engineer, is working on a patch to implement the third improvement and initialize a check for update automatically when ‘About Firefox’ opens – look for this addition in the next beta!

Note: Google Chrome already does something similar with its about window.  While these data driven recommendations were reached independently,  Chrome should be given credit for identifying this connection early-on.

Visualizing Firefox 4 Beta Usage

Last July, we presented initial analysis from our first comprehensive user interface (UI) study through an interactive, web-based heatmap. Many findings aligned with our expectations, but there were a few surprises. For example, only 12% of users clicked on the (tab bar) New Tab button and over 55% of users performed searches through the Location Bar.

Since then, we’ve introduced a major UI revision in Firefox 4 beta; tabs were moved on top, the Reload and Stop buttons were combined, and, more generally, the UI was simplified and streamlined.

Today, we’re assessing the impact of these changes through the first update to our heatmap. By intuitively presenting usability data on top of the Firefox UI, we aim to reduce speculation about Firefox usage and help designers make better informed product decisions, faster.

Here are just a few questions we can now answer:

1) Will users prefer tabs on top, or switch back to tabs on bottom?
2) Will Windows users smoothly transition to the combined Firefox menu button?
3) With fewer options to open new tabs, will use of the (tab bar) New Tab button rise?
4) Do advanced users interact with more of Firefox’s features, and, if so, which?

Heatmap Updates

Before I address these questions, let me briefly describe our heatmap updates. In addition to adjusting to the new UI, we:

1) Increased our sample size from just under 10,000 to well over 230,000 users (being integrated into the beta has its perks!).

2) Instrumented the combined Firefox menu button, doubling the number of UI elements items tracked.

3) Added bar charts that compare stats for beginner, intermediate, and advanced users. To view these charts, hover over any UI element.


Now, back to the questions.

1) Tabs on top: Initial analysis indicates that this move was well received by our users; 92% of users kept tabs on top for the majority of their browsing sessions.

2) Combined menu button: Collapsing the menu bar into a single button was one of most significant changes to the Firefox 4 UI. How did Windows 7 and Vista users react to the move?

The majority appear comfortable with this change, but 29.7% did switch back to the traditional menu bar. This relatively large number is not surprising because early betas included an incomplete version of the Firefox button. To access basic menu items, including Downloads, New Tab, and Start Private Browsing, users needed to revert to the traditional menu.

3) New tab button: In our first iteration of the Test Pilot UI study, we were surprised to see that only 12% of people used the (tab bar) New Tab button. But in this study, the New Tab button was the fourth most commonly used element, with over 88% of users clicking the button at least once.

This large shift can be partially accounted for by both the lack of a New Tab menu item in the Firefox button and the removal of the New Tab button from the Customize Toolbars pane. In early betas, the New Tab button in the tab bar was the only way to open a blank tab besides the keyboard shortcut “Ctrl + t”.

4) Tech level differences: The heatmap affirms many of the usage differences we expected to see across varying skill levels. More beginner users click on UI elements for common tasks that have popular shortcuts (e.g. Back, Forward, Bookmark Star). They also are more likely to use the Go button in the Location and Search Bars.

Advanced users, on the other hand, are more likely to use the Location Bar and search more frequently. They also are heavier users of the Tab Scroll arrows, the List All Tabs button, the RSS icon, and the Site Identity button — perhaps reflecting a better understanding of online security.


We’ve covered some basic findings, but are only beginning to dive into the data. For those that want to look at the raw numbers, we’ll post data samples soon (check back here).

As always, Mozilla does not make design decisions based on Test Pilot data alone. Despite increasing our sample size twenty-fold, our beta population is still not representative.

Additionally, quantitative data tracks how users interact the new UI, but now how they feel about specific design decisions. Aakash Desai has headed up some terrific work collecting qualitative data at So far, users’ response to the new user interface has been exceedingly positive!

Notice any thing unusual in the data or have another take on our analysis? Please leave your thoughts in the comments!

Who Are Our Firefox 4 Beta users?

As an intern here at Mozilla, I’ve learned that one of the core goals of the organization is to help build a better Internet by creating great software like the Firefox browser.  But like many other endeavors, one of the first steps to developing a browser, or software in general, is to know the end-users.

Consequently, we were excited to examine responses to the Firefox 4 Beta Background Survey. This short survey was pushed to all Beta users through the Feedback add-on and asked a few basic browsing behavior and demographic questions. In all, over 30,000 people, or roughly 7% of the entire Beta user-base, were kind enough to submit their responses.  Analyzing these submissions will not only help us understand the current Beta users, but also reveal the missing user groups we need to acquire to make the Beta sample more representative of the larger, general Firefox population.

So what did we find out? Who are our Firefox 4 Beta users?  Here are our main highlights:

1) The personal computer has been around longer than 75% of our Beta users

Although not surprising, our Beta population is concentrated among the lower age groups – over 75% of the Beta population is younger than 35 years old!  Even though we are skewed towards younger users, the good news is that there was a moderate increase in the participation of older users compared to our previous Test Pilot only sample.

2) Where are our Female Users? 96% of our Beta Population is male!

Evidently, our Beta Population is overwhelmingly skewed in terms of gender as 96% of users submitting feedback were male.

3) Early adopters are highly technical and spend multiple hours on the Internet a day

Due to the nature of betas, we expected the early adopters to be highly technical and spend a significant amount of time online.  Indeed, 69% of our Beta users are on the Web for over 4 hours a day, and 71% of gave themselves an advanced technical rating.

4) Entertainment, Communication and Socializing Drive Web Usage.

One of the most interesting aspects of this demographics survey was the potential to gain insights into the online interests of our Beta users.  We were particularly interested in identifying the main reasons why they use the web, as well as their most frequent visited websites. In particular, we found Beta users have three main reasons or using the web: Entertainment, Communication and Socializing.  We also noticed how these drivers were reflected in the most frequently visited websites as the top sites included News, Webmail, Social Networking, and Video content sites.

So what does all this survey information show? Overall, it clearly indicates that our Beta sample is far from representative of the general Firefox population. And in order to achieve more representative feedback, we need to increase the female, older and less technical user bases; we will definitely coordinate our Marketing, Metrics, and Community efforts to reach out to these users.

Maybe you existing Beta users can do some communicating and socializing to help us out! Tell your friends to download Firefox 4 Beta and help test the future of the Web!  Also, expect a follow-up blog post in the future as the makeup of the Beta sample evolves and (hopefully) becomes more representative.

Understanding Private Browsing

Private Browsing was introduced in Firefox 3.5, giving users the option of browsing the web without keeping track of their history.  A recent Test Pilot study recorded, among other things, the time users activated Private Browsing, and the time they deactivated it.  Though what happens in Private Browsing stays in Private Private Browsing – that is, neither Firefox nor Test Pilot records anything during the period – we did learn a few things about the timing and duration of of Private Browsing mode sessions.

Here are a few simple insights we’ve gleaned from the data.

Note: Test Pilot is an opt-in service for Firefox 4 Beta users.  We would never record a user’s activity for any reason unless they explicitly signed up for this study.  Again, we do not record any information at all during a user’s Private Browsing session – only when private browsing was entered and exited. You can read over the Test Pilot Privacy Policy here.

Activation spikes at lunch.

Though people switch into Private Browsing mode throughout the day, there are a few periods where activation surges:

  1. Lunch: users likely switch into Private Browsing during their lunch breaks.  We see a major spike between 11am and 2pm.
  2. After School / Work: users appear to switch on Private Browsing just after they’ve returned from work or school, which is around 5pm.
  3. After Dinner: there is another substantial usage peak between nine and ten pm.
  4. Late Night: a minor spike exists an hour or two after midnight.

The 10-minute window is the most common.

Now that we know when users jump into Private Browsing, the next question involves finding out how long users stay in it.  The 25th percentile stays on for about 4 and a half minutes, the 75th percentile around 22 minutes, and the median stays in for about 10 minutes.

This trend appears to hold over the entire course of the day, with the notable exception of 5pm.  For some reason the median and lower quantiles are lower than the other hours.

We’ve more insights to glean from the Week-In-The-Life study.  Stay tuned for more.

Firefox Main Window Study: A Heatmap Visualization

Last month, Mike Beltzner shared an early product plan for the next version of Firefox with the Mozilla community and presented our vision for Firefox 4. A big upgrade coming with Firefox 4 is the new theme project: creating a new, sleek, and simpler default theme for the browser. The UX team has been working tirelessly on this – streamlining and modernizing the user interface, particularly on Windows, given the aesthetic changes introduced by Vista and Windows 7.

Note: Plans May Change.

Metrics and Test Pilot have been doing what we can to help guide the UX team as they create an optimized design. A few months ago, Test Pilot released the Menu Item Usage Study, from which we gained valuable insight into how users interact with the menu bar. And a few weeks ago, we released the Main Window Usage Study to capture the remaining major UI elements of the browser, and provide more data-driven recommendations.

Today, we’d like to present some initial findings from this study. Like previous Test Pilot studies, the main window study ran for 5 days and covered Firefox versions 3.5 and up. The sample consists of 9,667 users who have opted into the Test Pilot program AND opted to submit their data.

We usually display study results in charts and graphs on this blog, but for this study, we were inspired by the work of principal designer, Alex Faaborg, and came up with a slightly different kind of visualization. Back in March, Alex created a heatmap to visualize the menu study data (his post is also a great example of how the UX team is using Test Pilot data to inform design decisions). Overlaying the actual UI and putting the data in context was really useful, so with his help, we expanded on the idea and constructed this interactive, web-based heatmap for the main window study:

(image links to heatmap)

Currently, the heatmap can display the usage data in two ways. Selecting ‘Used by x% of users’ displays the percentage of users who clicked each UI element at least one time, and ‘Clicks per user’ is simply the frequency click count for each item, divided by the number of users in the sample (we normalize for sample size to make comparisons across the different subsets more meaningful). The first approach is useful because it mitigates the effect of outliers, since each person who clicks an item counts only once, whether they clicked the item 1 time or 25,000 times. The obvious drawback is that this metric fails to capture intensity of use, which ‘Clicks per user’ does capture.

You can also choose to examine specific subsets within the sample; there are selectors to display breakdowns by OS and by self-reported computer/web skill level. Unfortunately, you cannot combine these subsets yet; that is, while you can look at data for just Windows users or for just Advanced users, you cannot look at data for the subset of Advanced, Windows users. Our sample is too small for such sub-setting right now, but we’ll add this ability as soon as our sample size increases.

Be sure not to miss the table to the right of the heatmap – it lists all the UI elements in descending order and also outlines the color spectrum. In addition, if you are viewing data for a subset, the table will show the differences between the currently selected subset and the other related subsets to make comparison easy (the differences for the ‘Used by x% of users’ filter are simply one subset minus the other, the differences for the ‘Clicks per user’ filter are in percentage change).

We’ll discuss results more thoroughly in future blog posts, but here are a few findings that immediately jumped out to us. As always, these findings apply to our sample of Test Pilot users only, and we realize this sample is not likely representative of the population of Firefox users as a whole. Furthermore, its important to remember that usage data like this is only one part of a rigorous design process, and that it alone will not dictate any design decisions.

  • The Back button is used far more often than any other navigation element (by which we mean the Back, Forward, Reload, Stop, and Home buttons). 93.1% of study participants used the Back button at least once, and on average, each user clicked Back 66.2 times over the 5 days – that’s 3x more clicks than the Reload button, 10x more than the Home button, and over 30x more than the Forward and Stop buttons!
  • Of all items that are not in the Firefox main window by default, the New Tab Button (Navigation Toolbar) is the most commonly used, both by % of users and frequency count.  Looking at click counts, the New Tab Button (Navigation Toolbar) is a more popular way to generate new tabs than the New Tab Button (Tab Bar) that is located in the tab strip by default.  The New Tab Button (Navigation Toolbar) is also more often used than the default UI elements Bookmark Star and RSS Icon by click count.
  • Another conspicuous result is the disparate use of the scroll arrows in comparison to the scroll slider.  This holds true for both the vertical scroll bar and the horizontal scroll bar; while 89% of people used the vertical scroll slider, only 21.2% and 16.6% of people used the up and down arrows. And although 42.8% of users used the horizontal scroll slider, only 2.1% and 0.9% clicked the left and right arrows to scroll.

Besides putting the data in a spatial context, another benefit of this visualization is that it is easy to reuse, update, and improve. We plan to add more data, more breakdowns, and the ability to synthesize multiple subsets in the near future. We also want to improve the robustness of our findings by filtering out certain extensions (e.g. some users have the Google Toolbar and might be searching through that channel instead of the integrated search box), and by increasing our sample size.

In addition, the heatmap will be updated over the course of the Firefox 4 beta program. Hopefully, this visualization will help us understand how the various UI changes affect user behavior, and ensure that these design decisions are in fact improving the product for our (beloved) end-users. Look for these updates soon!

Thanks again to Alex Faaborg, and to all Test Pilot users for opting-in to provide us with the data. If you’d like to help us understand how people use Firefox and build a better browser, head over to the Test Pilot website to learn more and download the add-on. And as always, please feel free to download the raw data and supplement our analysis with your own!

Firefox’s Adoption Funnel (II)

We wanted to share a quick update on a recent conversation regarding Firefox’s conversion and adoption funnel.  As an initial follow-up, Blake highlighted an experiment and a resulting set of simple speed changes to Mozilla’s web properties that will drive 60,000,000 incremental downloads of Firefox (annually).  And then just this week, we ran a different experiment – a test of download speeds, i.e., after a person clicks the Firefox download button, how does the wait time for getting the full file affect that person’s propensity to stick around and complete the download process?

We look forward to soon sharing the methodology, data, results, and implications from this latest experiment.  Stay tuned!

(image attributable to under a creative commons license.)