Menu Item Usage Study: An Update to the Initial Analysis

Now that we’ve ironed out some data issues and cleaned the data sets, we can get back to our analysis.  If you remember, we were trying to answer 3 questions as the UX team contemplates modifying the design of the menu bar for future Firefox (Windows) versions:

  • Which menu items are the most commonly used?
  • Which menu items are the least commonly used?
  • How long do users spend exploring the menu bar contents before selecting each particular menu item?

We took a stab at answering the first two questions by examining the total number of clicks on each menu item.  Again, all data is for mouse user interactions only–keyboard shortcuts will obviously remain consistent under any design.  The updated chart is shown below:

The most significant change is the usage of “Copy”, “Paste”, “Back”, and “Reload”.  These menu items were previously part of the top 10 most commonly utilized commands, but each item now accounts for less than .4% of total clicks.  This change isn’t too surprising since these commands are all available from context menus, and a major part of our data cleaning was the removal of context menu events that were incorrectly tracked as mouse clicks.

The revised ordering of these items also aligns more closely with how we conventionally believe people interact with the menu bar, especially for “Copy” and “Paste”.  Crtl + C and Ctrl + V are two of the most well known keyboard shortcuts, and intuitively, it seems unlikely that many users would forgo the hot keys and use the menu bar for these commands (especially Test Pilot users).

Besides these items, the relative order is fairly unchanged.  Notably, “User Bookmark Item” still dominants usage of the menu bar: 41% of all interactions are “User Bookmark Item” hits.

In previous posts, we also acknowledged that a major problem with analyzing aggregated data in this way is the potential for outliers to skew our results.  In order to present a more complete picture, we again move beyond looking at aggregated counts to examining how these counts are distributed. The table below presents some key statistics on the distribution of clicks for each menu item.

The distributions are now far less distorted by outliers, although clearly some right-skewness is still present. “User Bookmark Item” and “New Tab” are two of the more skewed distributions as both means are 4x greater than the medians.

Wrap Up
Now that we’ve revised our initial analysis, we will discuss new issues regarding menu usage behavior in the next series of posts. As always, thanks to all Test Pilot users and to everyone for the insightful comments–remember, more information on Test Pilot studies is available here, including the data samples themselves for anyone interested in doing their own analysis!

8 responses

  1. Zack wrote on :

    Could we get that table of distributions as boxplots, please? ‘s really easy with ggplot2.

  2. zombie wrote on :

    i hope you will now cancel the cut+copy+paste mini toolbar from the new themevv

  3. James John Malcolm wrote on :

    ” Again, all data is for mouse user interactions only–keyboard shortcuts will obviously remain consistent under any design.

    That’s not necessarily true. I’ve been using the awkward cmd+shift+T for undo tab close, because that item isn’t the top item in the tab-context menu (and the Menu Editor extension doesn’t work with minefield).

    On a related note, have you filtered out the people using Menu Editor? Their (context) menu’s are likely to be very different from other peoples. (Or are they counted ok?)

  4. Dan wrote on :

    It’s a shame there are so many “Unknown”. Also I think the high frequency of “Check for Updates” shows that this is quite highly skewed towards alpha, beta and nightly testers too. However, it’s still interesting.

  5. Kurt (supernova_00) wrote on :

    I’m pretty sure the “Unknown” are all extension menu items. I wonder why so many people use the “Bookmark this Page” entry instead of one-click on the star button. Also, the “Undo Closed tab” is the reason why I create the undo tabs closed button extension. Luckily the “New Tab” should disappear soon since the new tab button is now on the tabs toolbar.

  6. Christopher Jung wrote on :

    @Dan – Kurt (supernova_00) is right on. “Unknown” hits are all from menu items added by add-ons/extensions and unfortunately, we are unable to parse those out right now.

  7. Dan wrote on :

    Oh, OK, thanks. I assumed that “Add-Ons” in the graph was for all extension-created menu-items, but it seems a lot of people are actually clicking on Tools -> Add-ons. Another indication that this probably isn’t your average user group (although I know that the percentage of people using add-ons is higher than I originally thought).

  8. john wrote on :

    Thank you Christopher ,I’ve been using the awkward cmd+shift+T for undo tab close, because that item isn’t the top item in the tab-context menu