Menu Item Usage Study: Part I

mozilla

18 responses

For the last few days the Test Pilot team at Mozilla Labs has been running a test to explore usage of the Firefox menu bar. Ever since Mosaic 1.0 web browsers have had a standard menu bar–one that has always followed the design of a standard desktop publishing application, containing top level commands like File and Edit, even those these commands are not necessarily relevant to a web browser.

In order to streamline the Firefox user interface, and to match the overall interactive design of Windows 7, the Firefox UX team is exploring collapsing the menu bar into a single “application button” when Firefox is running on a modern version of Windows.

This menu item usage study will help guide the UX team as they create a fully optimized design by answering 3 questions.

  • Which menu items are the most commonly used?
  • Which menu items are the least commonly used?
  • How long do users spend exploring the menu bar contents before selecting each particular menu item?

In this post, we will discuss some preliminary findings regarding the first 2 of these 3 questions. Look for further analysis and a discussion of the 3rd question in our next post!

Experiment Results
The most obvious way to determine the most and least commonly used menu items is to simply aggregate the total number of menu item clicks for all users.

This graph shows just that, presenting each menu item’s relative use for all UI methods (both mouse and keyboard shortcuts). Even from this simple analysis, we can see some justification for a condensed toolbar as many of the items are used very infrequently compared to the other menu items. For example, the menu items from “Page Setup” to “Character Encoding/UTF-16” each make up less than 0.01% of the total menu bar clicks.

While looking at the total number of item clicks can be informative, since menu bars are designed for mouse use, it is more relevant to look at item usage for just the mouse UI method (excluding keyboard shortcuts).

Examining the data in this way presents a slightly different picture: the top 5 most commonly used menu items are now “User Bookmark Item”, “Copy”, “Paste”, “Add-Ons”, and “Back”. In addition to “Add-Ons”, “Options” and “Bookmark This Page” are newly part of the top 10, replacing “Find”, “Open Location”, and “Find Again”.

Again these changes simply result from eliminating keyboard shortcut clicks and help us distinguish between mouse driven menu items and keyboard driven items. For example, by comparing the mouse UI chart (right) with the original all UI chart (left) we can clearly see that “New Tab” and “Close Tab” are predominately driven by keyboard shortcuts (as expected) and may not be the two most critical items to a mouse oriented toolbar (as suggested by the original chart).

Another interesting approach to these questions is to group the items by menu and visualize the data in this form (again, data is just for Mouse UI).

This visualization presents information on two levels: the area of the circles are proportional to the total number of clicks for the menu group as a whole, and the slices correspond to the share of clicks for each item within the menu group. Bookmarks and Edit are by far the most utilized menus, representing over 70% of total clicks.

The high use of the bookmarks menu is somewhat surprising; an obvious problem of looking at aggregated data like this is the potential for outliers to skew the data. It will be interesting to delve into this issue more in depth and determine if the Bookmark menu (and other menus and menu items) is genuinely an important menu group for all users, or if the high usage is driven by a set of relatively few users who interact with the Bookmark menu extremely frequently.

Wrap Up
Next time we will take our analysis further and move from answering questions about the frequency of item usage to examining how long users spend exploring the menu bar before selecting each particular menu item.

Thanks again to the Test Pilot Team and to all Test Pilot users for providing us with the data. Remember more information on Test Pilot studies can be found here. Anyone interested can also download data samples for this and other Test Pilot Studies from the website!

18 responses

  1. sonny wrote on :

    Interesting.

  2. sonny wrote on :

    It would be interesting to know the source.
    I mean copy and past are the most common used action but do you take in account context menu? If yes I guess most of the time people use them from context menu on text field.

  3. Axel Hecht wrote on :

    Do you have data on locale and regional distribution of your test runs?

    The reason I’m asking is that I see the charset menu item “just” making it into the list, and I wonder if that’s depending on which part of the internet people are looking at.

  4. Agusum wrote on :

    Very interesting results. You should keep in mind, however, that Test Pilot users are usually techie people (myself included) and not like the average Joe. For example, I can’t imagine normal users using a keyboard shortcut to close a tab.

  5. Andyed wrote on :

    Would love to see the analysis code you use for this summary shared!

    I’ve posted my R source for pruning down the dataset to Menu 1 successful clicks and doing a regression by menu offset to time. http://etherpad.com/RUM4jIIEx5

    Pretty noisy! Next step: recode a local offset variable from menu offset and try the regression on other menus.

  6. dao wrote on :

    I’m slightly confused by your distinction between keyboard and mouse. Does keyboard mean shortcuts like Ctrl+C? If so, does it also include Alt+E, C?

  7. SilverWave wrote on :

    Copy and Paste – wow that looks odd.

    techies use ctrl+c cnrl+v … others select text and right click etc..

    Edit looks odd unless you are using Linux and go Edit > Preferences?

    I think it may be that we seeing a odd distribution because taken as a whole menu use is so low.

    As an example, how may users click the menu close item, instead of the top right X button.

    Guessing it could be 90% for top right X button but wouldn’t be surprised if its 99%…

    If so you are getting odd people doing odd things…

    Then you are looking at what they do.

    Odd things… QED 🙂

    Cheers very interesting indeed.

  8. Dan wrote on :

    I think this does include the context menus, but a clarification would be good.

    “or if the high usage is driven by a set of relatively few users who interact with the … menu extremely frequently”

    I think this is a good point. Perhaps you could look at the percentage of users that use a particular menu item rather than the percentage of clicks.

  9. Blake Cutler wrote on :

    Good suggestion Dan. We’re working on that now.

  10. Blake Cutler wrote on :

    @Andyed – Thanks for sharing your work! We plan on sharing our analysis code sometime in the next few weeks.

  11. Zaratustra wrote on :

    Maybe people use the ‘add new bookmark’ menu option so often because it doesn’t have a listed keyboard shortcut?

  12. jany wrote on :

    It is not as interesting as you describe it.
    My explanation (only for “CLOSE TAB” / “NEW TAB” problem):
    Firefox default settings (like unchecked “Hide the tab bar when only one tab is open”) caused this. Users learned how to open new (close) tab via menu, because there was not other user friendly approach to do this (hotkeys is not user friendly, although I prefer this) with only one tab open. And this approach is usable () to open/close tab all the time while you are browsing, so why change it. I know, there is a reason to change e.g. because of speed of browsing you can use hotkeys, but you can still use attested approach (via menu).

    My explanation (only for COPY problem):
    Why can I still select the text without any following action/operation like copy/paste? To copy some text I have to do minimal 3 moves – (select -> edit (in menu) -> copy (in menu)) or (select -> do right mouseclick -> copy (in contextmenu)). There should be an automation – When I select some text then context menu should appear automatically.

  13. jany wrote on :

    “unchecked” => “checked” (in my previous post)

  14. Dale wrote on :

    If you change the traditional menu bar many like myself will remove Firefox from our computers. I had Chrome installed just long enough to find out there was no way to show the menu bar, then I uninstalled it. Look at the angry feedback Chrome is getting about this issue. Many, many people have answered no menu bar no Chrome. Don’t make the same mistake.

  15. du hoc uc wrote on :

    Interesting.

  16. dien dan mua ban wrote on :

    This is really nice blog, I am very impressed.

  17. Quang ba web wrote on :

    thanks your information

  18. nerkn wrote on :

    I always thought i’m in the %0.01th percentage.
    When I need something it should be there.
    I think working for 3 months, I would definitly need to some property, if it is not there after my 10.000 click becomes absolote. I would get angry. I would think to move other software.
    %0.01 is nothing statistically but personally if im that number, it is important. I think features can be hidden but they should available, even if took 10 mouse clicks If it would worth, user clicks.
    I like the idea of having much more pixels.