Mozilla Open Data Visualization Competition – How Do People Use Firefox

Christopher Jung

1

The Mozilla Metrics Team, together with Mozilla Labs and the growing Mozilla Research Initiative, is excited to announce our first Open Data Visualization Competition.

Using data from Mozilla’s own open data program, Test Pilot, we’d like to explore creative visual answers to the question: “How do people use Firefox?” We are looking for compelling visualizations that tell detailed, meaningful and yet easy-to-interpret stories about interesting user activities.

Read on for details on the data (accessible Nov. 17), prizes, and judges, including special judge, David Smith from Revolution Analytics. Also, don’t forget to check out the official Competition page and follow @moztestpilot on Twitter for news and updates.

The Data

This competition is based on Mozilla’s own open data program, Test Pilot. Test Pliot is a user research platform that collecting structured user data through Firefox. All data is gathered through pre-defined Test Pilot studies which aim to explore how people use their web browser and the Internet.

Currently, over 1 million Firefox users from all over the world participate in Test Pilot studies. The goal for this platform is to encourage everyone from all skill levels to improve the Web experience by conducting and participating in these studies. Test Pilot study results are made available under open licenses, with the data being anonymized before release. (For more information about the Test Pilot data policy, please check Privacy Policy.)

For this challenge, we will use data from two recent Test Pilot studies:

Partners and Judges

We are honored to have David Smith from Revolution Analytics partnering with us and serving as a special judge. Members from Mozilla Labs and Mozilla Metrics will form the rest of the judging panel.

Prizes

To recognize the awesome work from participants, some great prizes are at stake:

  • Grand prize will be a $300 Amazon gift card
  • Two “Best in Class” teams will receive a set of Edward Tufte’s books

We’ll also present all submissions on the Test Pilot website and Mozilla Metrics blog in a special post to highlight your work.

Join the competition!

You can choose any tools you like for your analysis and visualization, including but not limited to: R, Matlab, Protovis, Processing or IBM many eyes. You can participate solo or team up with other people. You are also welcome to enter as many times as you like. If you are interested to join the competition, please follow the following important dates:

  • Nov. 17th: Check the official Competition page on Nov. 17th to download the data
  • Dec.5th : Go here to submit your results and enter the competition before Dec.5th.
  • Dec. 14th: Winning visualizations will be announced on Dec. 14th.

To facilitate the free exchange of ideas, all visualizations and other contributions you make to this challenge must be contributed under the Creative Commons Attribution-ShareAlike 3.0 license.

About the New ‘About Firefox’ Window

Christopher Jung

14

Earlier this year, Test Pilot ran a Menu Item Usage Study to better understand how users interact with the traditional menu bar as Firefox transitions to a more streamlined, single “application button.” We’ve already shared some analysis from this study: we answered which menu items are the most and least commonly used, explored the distribution of menu interactions, and learned that most users actually use very few commands from the menu bar.

We also examined the sequence of menu bar actions – specifically looking for common sequences in commands. The strongest connection was between the ‘About Firefox’ and ‘Check for Updates’ menu items; after clicking ‘About Firefox,’ 9% of users click ‘Check for Updates’ within 20 seconds.

From this data, we reasonably assume that many users go to “About Firefox” to check whether they should update. Accordingly, three minor changes to these menu items should improve user experience.

  1. Move ‘Check for Updates’ from the ‘Help’ header to the ‘Firefox’ header directly under ‘About Firefox’ so these related commands are logically grouped together (on Mac OSX).
  2. Add a ‘Check for Updates’ button directly to the ‘About Firefox’ window.
  3. If automatic updates are set, initialize a check for updates when ‘About Firefox’ opens, and give the user the option to update directly from the window (if any updates are available).

Luckily, we have some excellent engineers working on Firefox who came to the same conclusions.  Margaret Leibovic, one of these awesome engineers, recently blogged about her first month’s work here at Mozilla -  this work included implementing the first 2 of these changes.  Check out the new ‘About Firefox’ window below:

The relocation of ‘Check for Updates’ and this refreshed window have already landed in Minefield, and will be integrated into Firefox 4 from beta 7. Robert Strong, another Firefox Engineer, is working on a patch to implement the third improvement and initialize a check for update automatically when ‘About Firefox’ opens – look for this addition in the next beta!

Note: Google Chrome already does something similar with its about window.  While these data driven recommendations were reached independently,  Chrome should be given credit for identifying this connection early-on.

Visualizing Firefox 4 Beta Usage

Blake Cutler

11

Last July, we presented initial analysis from our first comprehensive user interface (UI) study through an interactive, web-based heatmap. Many findings aligned with our expectations, but there were a few surprises. For example, only 12% of users clicked on the (tab bar) New Tab button and over 55% of users performed searches through the Location Bar.

Since then, we’ve introduced a major UI revision in Firefox 4 beta; tabs were moved on top, the Reload and Stop buttons were combined, and, more generally, the UI was simplified and streamlined.

Today, we’re assessing the impact of these changes through the first update to our heatmap. By intuitively presenting usability data on top of the Firefox UI, we aim to reduce speculation about Firefox usage and help designers make better informed product decisions, faster.

Here are just a few questions we can now answer:

1) Will users prefer tabs on top, or switch back to tabs on bottom?
2) Will Windows users smoothly transition to the combined Firefox menu button?
3) With fewer options to open new tabs, will use of the (tab bar) New Tab button rise?
4) Do advanced users interact with more of Firefox’s features, and, if so, which?

Heatmap Updates

Before I address these questions, let me briefly describe our heatmap updates. In addition to adjusting to the new UI, we:

1) Increased our sample size from just under 10,000 to well over 230,000 users (being integrated into the beta has its perks!).

2) Instrumented the combined Firefox menu button, doubling the number of UI elements items tracked.

3) Added bar charts that compare stats for beginner, intermediate, and advanced users. To view these charts, hover over any UI element.

Analysis

Now, back to the questions.

1) Tabs on top: Initial analysis indicates that this move was well received by our users; 92% of users kept tabs on top for the majority of their browsing sessions.

2) Combined menu button: Collapsing the menu bar into a single button was one of most significant changes to the Firefox 4 UI. How did Windows 7 and Vista users react to the move?

The majority appear comfortable with this change, but 29.7% did switch back to the traditional menu bar. This relatively large number is not surprising because early betas included an incomplete version of the Firefox button. To access basic menu items, including Downloads, New Tab, and Start Private Browsing, users needed to revert to the traditional menu.

3) New tab button: In our first iteration of the Test Pilot UI study, we were surprised to see that only 12% of people used the (tab bar) New Tab button. But in this study, the New Tab button was the fourth most commonly used element, with over 88% of users clicking the button at least once.

This large shift can be partially accounted for by both the lack of a New Tab menu item in the Firefox button and the removal of the New Tab button from the Customize Toolbars pane. In early betas, the New Tab button in the tab bar was the only way to open a blank tab besides the keyboard shortcut “Ctrl + t”.

4) Tech level differences: The heatmap affirms many of the usage differences we expected to see across varying skill levels. More beginner users click on UI elements for common tasks that have popular shortcuts (e.g. Back, Forward, Bookmark Star). They also are more likely to use the Go button in the Location and Search Bars.

Advanced users, on the other hand, are more likely to use the Location Bar and search more frequently. They also are heavier users of the Tab Scroll arrows, the List All Tabs button, the RSS icon, and the Site Identity button — perhaps reflecting a better understanding of online security.

Conclusion

We’ve covered some basic findings, but are only beginning to dive into the data. For those that want to look at the raw numbers, we’ll post data samples soon (check back here).

As always, Mozilla does not make design decisions based on Test Pilot data alone. Despite increasing our sample size twenty-fold, our beta population is still not representative.

Additionally, quantitative data tracks how users interact the new UI, but now how they feel about specific design decisions. Aakash Desai has headed up some terrific work collecting qualitative data at input.mozilla.com. So far, users’ response to the new user interface has been exceedingly positive!

Notice any thing unusual in the data or have another take on our analysis? Please leave your thoughts in the comments!

Who Are Our Firefox 4 Beta users?

Andres Garcia

7

As an intern here at Mozilla, I’ve learned that one of the core goals of the organization is to help build a better Internet by creating great software like the Firefox browser.  But like many other endeavors, one of the first steps to developing a browser, or software in general, is to know the end-users.

Consequently, we were excited to examine responses to the Firefox 4 Beta Background Survey. This short survey was pushed to all Beta users through the Feedback add-on and asked a few basic browsing behavior and demographic questions. In all, over 30,000 people, or roughly 7% of the entire Beta user-base, were kind enough to submit their responses.  Analyzing these submissions will not only help us understand the current Beta users, but also reveal the missing user groups we need to acquire to make the Beta sample more representative of the larger, general Firefox population.

So what did we find out? Who are our Firefox 4 Beta users?  Here are our main highlights:

1) The personal computer has been around longer than 75% of our Beta users

Although not surprising, our Beta population is concentrated among the lower age groups – over 75% of the Beta population is younger than 35 years old!  Even though we are skewed towards younger users, the good news is that there was a moderate increase in the participation of older users compared to our previous Test Pilot only sample.

2) Where are our Female Users? 96% of our Beta Population is male!

Evidently, our Beta Population is overwhelmingly skewed in terms of gender as 96% of users submitting feedback were male.

3) Early adopters are highly technical and spend multiple hours on the Internet a day

Due to the nature of betas, we expected the early adopters to be highly technical and spend a significant amount of time online.  Indeed, 69% of our Beta users are on the Web for over 4 hours a day, and 71% of gave themselves an advanced technical rating.

4) Entertainment, Communication and Socializing Drive Web Usage.


One of the most interesting aspects of this demographics survey was the potential to gain insights into the online interests of our Beta users.  We were particularly interested in identifying the main reasons why they use the web, as well as their most frequent visited websites. In particular, we found Beta users have three main reasons or using the web: Entertainment, Communication and Socializing.  We also noticed how these drivers were reflected in the most frequently visited websites as the top sites included News, Webmail, Social Networking, and Video content sites.

So what does all this survey information show? Overall, it clearly indicates that our Beta sample is far from representative of the general Firefox population. And in order to achieve more representative feedback, we need to increase the female, older and less technical user bases; we will definitely coordinate our Marketing, Metrics, and Community efforts to reach out to these users.

Maybe you existing Beta users can do some communicating and socializing to help us out! Tell your friends to download Firefox 4 Beta and help test the future of the Web!  Also, expect a follow-up blog post in the future as the makeup of the Beta sample evolves and (hopefully) becomes more representative.

Understanding Private Browsing

Hamilton Ulmer

20

Private Browsing was introduced in Firefox 3.5, giving users the option of browsing the web without keeping track of their history.  A recent Test Pilot study recorded, among other things, the time users activated Private Browsing, and the time they deactivated it.  Though what happens in Private Browsing stays in Private Private Browsing – that is, neither Firefox nor Test Pilot records anything during the period – we did learn a few things about the timing and duration of of Private Browsing mode sessions.

Here are a few simple insights we’ve gleaned from the data.

Note: Test Pilot is an opt-in service for Firefox 4 Beta users.  We would never record a user’s activity for any reason unless they explicitly signed up for this study.  Again, we do not record any information at all during a user’s Private Browsing session – only when private browsing was entered and exited. You can read over the Test Pilot Privacy Policy here.

Activation spikes at lunch.

Though people switch into Private Browsing mode throughout the day, there are a few periods where activation surges:

  1. Lunch: users likely switch into Private Browsing during their lunch breaks.  We see a major spike between 11am and 2pm.
  2. After School / Work: users appear to switch on Private Browsing just after they’ve returned from work or school, which is around 5pm.
  3. After Dinner: there is another substantial usage peak between nine and ten pm.
  4. Late Night: a minor spike exists an hour or two after midnight.

The 10-minute window is the most common.

Now that we know when users jump into Private Browsing, the next question involves finding out how long users stay in it.  The 25th percentile stays on for about 4 and a half minutes, the 75th percentile around 22 minutes, and the median stays in for about 10 minutes.

This trend appears to hold over the entire course of the day, with the notable exception of 5pm.  For some reason the median and lower quantiles are lower than the other hours.

We’ve more insights to glean from the Week-In-The-Life study.  Stay tuned for more.

Firefox Main Window Study: A Heatmap Visualization

Christopher Jung

13

Last month, Mike Beltzner shared an early product plan for the next version of Firefox with the Mozilla community and presented our vision for Firefox 4. A big upgrade coming with Firefox 4 is the new theme project: creating a new, sleek, and simpler default theme for the browser. The UX team has been working tirelessly on this – streamlining and modernizing the user interface, particularly on Windows, given the aesthetic changes introduced by Vista and Windows 7.

Note: Plans May Change.

Metrics and Test Pilot have been doing what we can to help guide the UX team as they create an optimized design. A few months ago, Test Pilot released the Menu Item Usage Study, from which we gained valuable insight into how users interact with the menu bar. And a few weeks ago, we released the Main Window Usage Study to capture the remaining major UI elements of the browser, and provide more data-driven recommendations.

Today, we’d like to present some initial findings from this study. Like previous Test Pilot studies, the main window study ran for 5 days and covered Firefox versions 3.5 and up. The sample consists of 9,667 users who have opted into the Test Pilot program AND opted to submit their data.

We usually display study results in charts and graphs on this blog, but for this study, we were inspired by the work of principal designer, Alex Faaborg, and came up with a slightly different kind of visualization. Back in March, Alex created a heatmap to visualize the menu study data (his post is also a great example of how the UX team is using Test Pilot data to inform design decisions). Overlaying the actual UI and putting the data in context was really useful, so with his help, we expanded on the idea and constructed this interactive, web-based heatmap for the main window study:


(image links to heatmap)

Currently, the heatmap can display the usage data in two ways. Selecting ‘Used by x% of users’ displays the percentage of users who clicked each UI element at least one time, and ‘Clicks per user’ is simply the frequency click count for each item, divided by the number of users in the sample (we normalize for sample size to make comparisons across the different subsets more meaningful). The first approach is useful because it mitigates the effect of outliers, since each person who clicks an item counts only once, whether they clicked the item 1 time or 25,000 times. The obvious drawback is that this metric fails to capture intensity of use, which ‘Clicks per user’ does capture.

You can also choose to examine specific subsets within the sample; there are selectors to display breakdowns by OS and by self-reported computer/web skill level. Unfortunately, you cannot combine these subsets yet; that is, while you can look at data for just Windows users or for just Advanced users, you cannot look at data for the subset of Advanced, Windows users. Our sample is too small for such sub-setting right now, but we’ll add this ability as soon as our sample size increases.

Be sure not to miss the table to the right of the heatmap – it lists all the UI elements in descending order and also outlines the color spectrum. In addition, if you are viewing data for a subset, the table will show the differences between the currently selected subset and the other related subsets to make comparison easy (the differences for the ‘Used by x% of users’ filter are simply one subset minus the other, the differences for the ‘Clicks per user’ filter are in percentage change).

We’ll discuss results more thoroughly in future blog posts, but here are a few findings that immediately jumped out to us. As always, these findings apply to our sample of Test Pilot users only, and we realize this sample is not likely representative of the population of Firefox users as a whole. Furthermore, its important to remember that usage data like this is only one part of a rigorous design process, and that it alone will not dictate any design decisions.

  • The Back button is used far more often than any other navigation element (by which we mean the Back, Forward, Reload, Stop, and Home buttons). 93.1% of study participants used the Back button at least once, and on average, each user clicked Back 66.2 times over the 5 days – that’s 3x more clicks than the Reload button, 10x more than the Home button, and over 30x more than the Forward and Stop buttons!
  • Of all items that are not in the Firefox main window by default, the New Tab Button (Navigation Toolbar) is the most commonly used, both by % of users and frequency count.  Looking at click counts, the New Tab Button (Navigation Toolbar) is a more popular way to generate new tabs than the New Tab Button (Tab Bar) that is located in the tab strip by default.  The New Tab Button (Navigation Toolbar) is also more often used than the default UI elements Bookmark Star and RSS Icon by click count.
  • Another conspicuous result is the disparate use of the scroll arrows in comparison to the scroll slider.  This holds true for both the vertical scroll bar and the horizontal scroll bar; while 89% of people used the vertical scroll slider, only 21.2% and 16.6% of people used the up and down arrows. And although 42.8% of users used the horizontal scroll slider, only 2.1% and 0.9% clicked the left and right arrows to scroll.

Besides putting the data in a spatial context, another benefit of this visualization is that it is easy to reuse, update, and improve. We plan to add more data, more breakdowns, and the ability to synthesize multiple subsets in the near future. We also want to improve the robustness of our findings by filtering out certain extensions (e.g. some users have the Google Toolbar and might be searching through that channel instead of the integrated search box), and by increasing our sample size.

In addition, the heatmap will be updated over the course of the Firefox 4 beta program. Hopefully, this visualization will help us understand how the various UI changes affect user behavior, and ensure that these design decisions are in fact improving the product for our (beloved) end-users. Look for these updates soon!

Thanks again to Alex Faaborg, and to all Test Pilot users for opting-in to provide us with the data. If you’d like to help us understand how people use Firefox and build a better browser, head over to the Test Pilot website to learn more and download the add-on. And as always, please feel free to download the raw data and supplement our analysis with your own!

Firefox’s Adoption Funnel (II)

Ken Kovash

3

We wanted to share a quick update on a recent conversation regarding Firefox’s conversion and adoption funnel.  As an initial follow-up, Blake highlighted an experiment and a resulting set of simple speed changes to Mozilla’s web properties that will drive 60,000,000 incremental downloads of Firefox (annually).  And then just this week, we ran a different experiment – a test of download speeds, i.e., after a person clicks the Firefox download button, how does the wait time for getting the full file affect that person’s propensity to stick around and complete the download process?

We look forward to soon sharing the methodology, data, results, and implications from this latest experiment.  Stay tuned!

(image attributable to http://www.flickr.com/photos/sarahbelle1/ under a creative commons license.)

Menu Item Usage Study: The 80-20 Rule?

Christopher Jung

16

Last post, we identified the most and least commonly used menu items. Today, we will answer another question: how many different menu items do people actually use? It’s a simple question, but one that could be critical to the proposal to condense the menu bar into a single application button.

The 80-20 rule, or the Pareto principle, suggests that the number is on the low end — around 14 items. This popular rule of thumb states that for many events, 80% of effects come from 20% of the causes. Applied to software, the common saying is that 80% of all end users generally use only 20% of an application’s features.

Let’s see if menu bar interactions follow a similar pattern:

We expected a median around 14, but the actual median is 3 — almost 5x smaller! This means that 50% of all users clicked 3 items or less during the course of the study. Moreover, only 25% of people clicked on more than 4 commands. The rule of thumb states that 80% of users use only 20% of an application’s features, but menu bar behavior is far more limited: 80% of users use only 5 menu items, or just 7% of the total number of commands!

We were also surprised by the number of people who click only one unique menu item. Considering that over 20% of people fall into this category, we want to investigate further and identify which single item these individuals are using. Identifying these items is especially important since these single commands account for 100% of these users’ menu bar interactions — imprudently removing any of these commands could severely damage the browsing experience of these users.

Conveniently, the list roughly matches the ranking of the most commonly used menu items. This means that we can continue to base menu design decisions off of this ranking, without crippling the single-item users. If the orders were radically different, we may have needed to give special consideration to items that are important to single-item users, but not commonly used otherwise.

The presence of items such as “User Bookmark Item” and “Unknown”, however, leads us to some qualification of our findings:

  • A few items, “User Bookmark Item”, “Unknown”, and “User History Item”, actually encapsulate more than one unique command.  We either did not find it useful to parse these items further (in the case of “User Bookmark Item” and “User History Item”), or were not able to (in the case of “Unknown”).
  • A few items were not tracked during the study due to a bug (Special Characters, Character Encoding/More Encodings/*, Character Encoding/Customize List, Show All History, Organize Bookmarks, Minimize, and Zoom).  This means that the number of unique items for users hitting these commands is slightly downward biased.
  • The study ran for 5 days as the goal was to learn about conventional menu bar behavior.  Over the course of a browser’s lifetime, users probably click a few more menu items, but we want to optimize the bar for normal usage, so limiting the scope to a typical work week is fairly reasonable.
  • The findings refer solely to mouse interactions — again, keyboard shortcuts will remain consistent under any design.
  • And as always, our sample is of Test Pilot users only, and is likely not representative of the population of Firefox users as a whole.

Nevertheless, we are confident in our general point that relatively few menu items account for nearly all the interactions with the menu bar. Given how many items are actually used, the menu bar is too comprehensive and takes up too much valuable real estate — in other words, a condensed menu bar seems well justified.

Thoughts? How many different menu items do you typically use?

Why People Don’t Upgrade Their Browser – Part IV

Christopher Jung

51

Web technology moves quickly, and we at Mozilla do our best to balance our values of keeping users safe and secure with also ensuring that we are giving users the ability to make choices about the software on their computers. We can’t keep up with the cost of providing security updates to older versions forever, though, so we’ll often try to encourage users to migrate to the latest version. A few weeks ago, we made another such offer to our Firefox 3 users, explaining that we were not planning on supporting that release anymore and asking them to upgrade to Firefox 3.6:

In the past, soliciting feedback from users has helped us understand why users were opting to not upgrade, so this time around, people who clicked on the “No thanks” option above were directed to a survey. Over 40,000 people were kind enough to share their thoughts with us.

So what do the results look like this time?

Let’s start by taking a look at responses to the first question:

Over half of the respondents stated that they were simply content with Firefox 3. And compared to what we saw last survey, far fewer users selected “Other reason.” Still, a quarter of users took the time to give an alternative explanation, so exploring these answers can definitely help us gain additional insight. A list of the most common phrases entered into the “Other reason” text field is presented below:

Two things immediately jump out:

  • First, confusion over cost has virtually disappeared — this had been a problem previously.
  • Second, the vast majority of comments are now about a lack of time.

The lack of comments about costs is a pleasant surprise. Acting on feedback from the last survey, we revised the update prompt, highlighting that the upgrade is free in two places. It seems that this simple addition was enough to eliminate most of the confusion.

The update prompt itself may also partly account for the rise in “no time” comments. With prior upgrade initiatives, users saw an advertisement and button similar to what you see below:

Users could then click “Get the new version” to initiate a 30-second update procedure. This time, however, instead of seeing a typical software dialog box, people encountered the update prompt via the Firefox “Whats new” page and were presented with the customary green Firefox download button.  Hitting the button also initiated the update, but via the more involved process of downloading and installing a fresh version of the browser.

Pushing out the update in this way was a one-time situation, and we’ll revert to the usual process and software dialog box in the future. To alleviate this issue further, we should also perhaps add to the prompt some indication that updating Firefox is relatively quick and painless.

Next, lets turn to Question 2. This was a free form text box so we manually read through a random sample of 10% of the responses and parsed them into categories.

Firefox 3.6’s compatibility, both with add-ons and with specific websites/applications, remains a key issue (although user perception might be playing a small role as well). On the other hand, general stability and performance issues are cited considerably less often: crash and speed comments together account for 11% of responses, down from the 25% we saw last survey.

Our two main insights from above are also evident here. Cost comments have essentially disappeared from Question 2 as well, although they previously made up 7% of responses. And again, time concerns have become a real sticking point for users.

The rise of these “no time” responses, coupled with the still considerable “too many updates” category, has triggered suggestions that extend beyond prompt and wording revisions. For example, some propose that updates occur automatically in the background unbeknownst to the user (à la chrome), perhaps with an easy option to downgrade (unlike chrome).

Clearly, we are a long ways from making any major changes to the update process, but we’d love to hear your thoughts on any of this!