Visualizing Firefox 4 Beta Usage

blakecutler

11 responses

Last July, we presented initial analysis from our first comprehensive user interface (UI) study through an interactive, web-based heatmap. Many findings aligned with our expectations, but there were a few surprises. For example, only 12% of users clicked on the (tab bar) New Tab button and over 55% of users performed searches through the Location Bar.

Since then, we’ve introduced a major UI revision in Firefox 4 beta; tabs were moved on top, the Reload and Stop buttons were combined, and, more generally, the UI was simplified and streamlined.

Today, we’re assessing the impact of these changes through the first update to our heatmap. By intuitively presenting usability data on top of the Firefox UI, we aim to reduce speculation about Firefox usage and help designers make better informed product decisions, faster.

Here are just a few questions we can now answer:

1) Will users prefer tabs on top, or switch back to tabs on bottom?
2) Will Windows users smoothly transition to the combined Firefox menu button?
3) With fewer options to open new tabs, will use of the (tab bar) New Tab button rise?
4) Do advanced users interact with more of Firefox’s features, and, if so, which?

Heatmap Updates

Before I address these questions, let me briefly describe our heatmap updates. In addition to adjusting to the new UI, we:

1) Increased our sample size from just under 10,000 to well over 230,000 users (being integrated into the beta has its perks!).

2) Instrumented the combined Firefox menu button, doubling the number of UI elements items tracked.

3) Added bar charts that compare stats for beginner, intermediate, and advanced users. To view these charts, hover over any UI element.

Analysis

Now, back to the questions.

1) Tabs on top: Initial analysis indicates that this move was well received by our users; 92% of users kept tabs on top for the majority of their browsing sessions.

2) Combined menu button: Collapsing the menu bar into a single button was one of most significant changes to the Firefox 4 UI. How did Windows 7 and Vista users react to the move?

The majority appear comfortable with this change, but 29.7% did switch back to the traditional menu bar. This relatively large number is not surprising because early betas included an incomplete version of the Firefox button. To access basic menu items, including Downloads, New Tab, and Start Private Browsing, users needed to revert to the traditional menu.

3) New tab button: In our first iteration of the Test Pilot UI study, we were surprised to see that only 12% of people used the (tab bar) New Tab button. But in this study, the New Tab button was the fourth most commonly used element, with over 88% of users clicking the button at least once.

This large shift can be partially accounted for by both the lack of a New Tab menu item in the Firefox button and the removal of the New Tab button from the Customize Toolbars pane. In early betas, the New Tab button in the tab bar was the only way to open a blank tab besides the keyboard shortcut “Ctrl + t”.

4) Tech level differences: The heatmap affirms many of the usage differences we expected to see across varying skill levels. More beginner users click on UI elements for common tasks that have popular shortcuts (e.g. Back, Forward, Bookmark Star). They also are more likely to use the Go button in the Location and Search Bars.

Advanced users, on the other hand, are more likely to use the Location Bar and search more frequently. They also are heavier users of the Tab Scroll arrows, the List All Tabs button, the RSS icon, and the Site Identity button — perhaps reflecting a better understanding of online security.

Conclusion

We’ve covered some basic findings, but are only beginning to dive into the data. For those that want to look at the raw numbers, we’ll post data samples soon (check back here).

As always, Mozilla does not make design decisions based on Test Pilot data alone. Despite increasing our sample size twenty-fold, our beta population is still not representative.

Additionally, quantitative data tracks how users interact the new UI, but now how they feel about specific design decisions. Aakash Desai has headed up some terrific work collecting qualitative data at input.mozilla.com. So far, users’ response to the new user interface has been exceedingly positive!

Notice any thing unusual in the data or have another take on our analysis? Please leave your thoughts in the comments!

11 responses

  1. Jan-Christoph Borchardt wrote on :

    Wow, another great data map! 🙂

    Just like Bill Burcham already mentioned in the previous test post, it would be really interesting to see the data for keyboard shortcuts. This came to my mind specifically when I saw that the low usage of »Find in This Page …«. Ctrl+W and Ctrl+T would certainly be interesting as well.

    Also, what about the menu items that have no data overlay?

  2. John wrote on :

    Great Work.

  3. Damian wrote on :

    Notice how moving mouse click actions to the far right of the browser basically destroys discoverability and usability.

    This has long been observed and implied in previous studies Mozilla has done or commisioned, but Firefox developers seem to ignore this fact and put really common features there : (

  4. Dan wrote on :

    “In early betas, the New Tab button in the tab bar was the only way to open a blank tab besides the keyboard shortcut “Ctrl + t”.”

    Double-clicking or middle-clicking the tab bar.

  5. Ankit wrote on :

    “Double-clicking or middle-clicking the tab bar.”

    I don’t think this is possible any longer. Or at least if it is, I can’t figure out how. When I try, I get a resized Firefox window for double click and no effect for middle click.

  6. John Orford wrote on :

    I was a bit peeved at there being no newtab available from customise, but the double click is even better and also new to me (thank you ankit). The more choices the better though surely?
    Like tabs on top, like FIrefox 4.

  7. yieph wrote on :

    great

  8. jeff wrote on :

    I miss the new tab button in 4. I’ve discovered that double clicking to the right of the open tab still works though, so I’ll have to learn to live with that I guess. I must be one of the 12% who actually used it, but I used it A LOT.

    I’d appreciate a new tab button as an option, Did I already say I REALLY miss it?

  9. Marc Vandenhende wrote on :

    I need the new tab button back! I have the “always show the tab bar” option off, to have more screen space available when I don’t need multiple web pages opened at once. But then there’s no tab bar button to click on, to open an additional tab if needed.

  10. Jonny wrote on :

    The only thing that’s bugging me with the new UI is that the drop down menu to go back several pages is gone which is really annoying having to go back through all the pages just to get to the one i wanted or go through my whole history, apart from that no problems so far.

  11. Mike wrote on :

    Jonny,
    I just got the Beta 8 version, and when I saw your comment I went to see for myself.

    It turns out you can right-click on the forward/back button to get the “drop down” menu of all pages in the stack, so that you can select which one to go to.