Now that we’ve ironed out some data issues and cleaned the data sets, we can get back to our analysis.  If you remember, we were trying to answer 3 questions as the UX team contemplates modifying the design of the menu bar for future Firefox (Windows) versions:

• Which menu items are the most commonly used?
• Which menu items are the least commonly used?
• How long do users spend exploring the menu bar contents before selecting each particular menu item?

We took a stab at answering the first two questions by examining the total number of clicks on each menu item.  Again, all data is for mouse user interactions only–keyboard shortcuts will obviously remain consistent under any design.  The updated chart is shown below:

The most significant change is the usage of “Copy”, “Paste”, “Back”, and “Reload”.  These menu items were previously part of the top 10 most commonly utilized commands, but each item now accounts for less than .4% of total clicks.  This change isn’t too surprising since these commands are all available from context menus, and a major part of our data cleaning was the removal of context menu events that were incorrectly tracked as mouse clicks.

The revised ordering of these items also aligns more closely with how we conventionally believe people interact with the menu bar, especially for “Copy” and “Paste”.  Crtl + C and Ctrl + V are two of the most well known keyboard shortcuts, and intuitively, it seems unlikely that many users would forgo the hot keys and use the menu bar for these commands (especially Test Pilot users).

Besides these items, the relative order is fairly unchanged.  Notably, “User Bookmark Item” still dominants usage of the menu bar: 41% of all interactions are “User Bookmark Item” hits.

In previous posts, we also acknowledged that a major problem with analyzing aggregated data in this way is the potential for outliers to skew our results.  In order to present a more complete picture, we again move beyond looking at aggregated counts to examining how these counts are distributed. The table below presents some key statistics on the distribution of clicks for each menu item.

The distributions are now far less distorted by outliers, although clearly some right-skewness is still present. “User Bookmark Item” and “New Tab” are two of the more skewed distributions as both means are 4x greater than the medians.

Wrap Up
Now that we’ve revised our initial analysis, we will discuss new issues regarding menu usage behavior in the next series of posts. As always, thanks to all Test Pilot users and to everyone for the insightful comments–remember, more information on Test Pilot studies is available here, including the data samples themselves for anyone interested in doing their own analysis!