Test Pilot New Tab Study Results

Lilian Weng

14

The new tab page in Firefox is intentionally left blank, while some browsers present rich information on a newly opened tab.

screenshots of new tab pages in different browsers

The decision to leave new tab pages in Firefox blank was driven, in part, by a suspicion that too much information in the new tab may distract users from getting to the destination intended for the new tab. To test whether this suspicion is true and to learn more about user behavior after opening a new tab to help redesign the Firefox new tab page, Test Pilot recently released the New Tab Study and will release a further test on multivariate design on new tab page soon.Test Pilot is a platform collecting structured user feedback through Firefox. It currently has about 3 millions users and all the studies are opt in. You can help us better understand how people use their web browser and the Internet in order to build better products by participating studies. Test Pilot add-on is available here.The study ran for 5 days and in all, we collected 256,282 valid submissions.
Results of the study show that on average each user daily:
  • opens 11 new blank tabs
  • loads 7 pages
  • visits 2 unique domains
  • visits 2 pages in a new tab before they leave or close it

Below are details on how a user loads a page in a new tab, their intentions when opening a new tab, and time spent on new tabs below.

How do users load a page in new tabs?

 

We detected 11 different methods to load a Web page in a blank tab page. Actions in the Url bar include pressing ENTER through keyboard, clicking the go button on the right side of the bar, clicking the Web page suggestions in the dropdown menu and pressing ENTER key for dropdown suggestions. Similarly 4 actions can be performed in the search bar too. Users can load a previously saved page from the bookmark bar in the toolbar or Bookmark/History in the menu bar.

Note:

  • The URL bar is most used when navigating to new websites.
  • The Search bar is also popular. Users rarely use search bar dropdown to look for old search terms.
  • The Bookmark toolbar is used more often than the bookmark menu button.
  • The History Menu button is seldom used.

We can also classify all methods for loading web pages into either keyboard-based or mouse-based category. Generally speaking, users have a slight preference for mouse usage.

 

Why do users open new tabs?

1.    Are they looking for a specific URL?

13.95% of new tabs (13,941,404) are opened while the text in the clipboard starts with “http” or “www”, which are very likely to be URL strings. The number is surprisingly high, although it may be caused by previous actions rather than by pasting for loading a specific URL.

2.    Users browse a limited set of domains, and only a small proportion of domains attract most visits

If we represent each user as a single point in the plot where x-axis is the number of pageloads, and y-axis is the number of unique domains visited, we can get the following graph. The dash line (diagonal) is what will happen if users always visit a different domain for each page load. When the users are not so active, pageloads less or around a few hundreds, the number of unique domains grows linearly. However, once users get to browse more, distinct domains tend to be stabilized and saturated.

Globally, we check the visit frequencies of all domains, and find that globally only 17.38% domains (461,133 unique domains in total) take 80% of the total page loads (8,291,541 pageloads in total). It verifies the famous “20-80” law of long tail phenomena.

 

On the individual level, we are interested in whether a single user performs the browsing movements according to the 20-80 law. For each individual, domains taking 80% of the total page visits is defined as “main domains”. A user can confirm the 20-80 law if the ratio of the number of his main domains to the number of distinct domains is around 20%. According to the following fig., active users browse more web pages everyday, but the number of primary sites they go to decreases proportionally. It suggests that when users visit more sites, they prefer to go to the same sites more frequently. The result supports the existence of a speed-dial new tab page to some extent.

 

Time Spent on New Tabs

According to the study results, on average, users open 2 pages in a new tab before they leave or close it. They load the first web page in 6 seconds (median) after they open new tabs, and stay on the tab for 1 minute (median) once they start browsing. The distributions of these two types of reaction timings display broad tails. The actually mean values are much higher than the medians: users load the first web page in 45 seconds (mean) after they open new tabs, and stay on the tab for 7 minute (mean) once they start browsing, since the outliners and expected noises can vary the mean value a lot.
Meanwhile, how users open a new tab can distinguish 2 groups of mouse-based users and keyboard-based users. The tabs invoked by “Plus Button” and “Double Click on TabBar” represent the group of mouse-based users, and the tabs invoked by “Command+T” represent the the group of keyboard-based users. The results turn out that keyboard-based users act slightly faster than mouse-based ones, and they can stay on the same new tab a bit longer.
The study is preliminary study for redesign requirement of the new tab pages in Firefox. We detect user behavior patterns of how they use the new tabs, including how they load a new page, broadness of domain visited, and the timing of different actions. In the following New Tab Multivariate Test, we will do a comparison between several designs of the new tab page, and more research questions will be answered, including whether too much information in the new tab may distract users from the original target or not.

14 responses

Post a comment

  1. Pingback from Test Pilot New Tab Study Results « Blog of Metrics on ::

    [...] [Cross-posted at Mozilla User Research] [...]

    Reply

  2. aelilea wrote on :

    Nice analysis!

    Did you also test how often the search bar is used directly after opening a new tab vs while already on another page? (Personally I have the suspicion the search bar could be integrated into a new tab page, because at least for me I mainly use it directly after Ctrl+T)

    Reply

    1. Lilian Weng wrote on ::

      nice catch! That actually can be done.
      I will run the analysis, and the result could be petty interesting :)

      Reply

  3. Bev wrote on :

    FWIW, I only open up a blank tab (using controlT ALWAYS) when I can’t click on a URL and have to cut+paste. The recent history displayed was not only distracting but annoying because I felt compelled to see what was there. There is NO reason to open a blank tab to look at something in recent history or tab closure — there are other methods of doing that which people are already used to.

    Reply

  4. Mads wrote on :

    I just started the new tab test pilot and I’ve found that the new tab page loads very slowly. Basically I have entered my destination URL and hit enter before the icons show. Moreover, it has failed to gather icons from URL’s such as http://facebook.com and http://www.google.com/reader/

    I know this test is to mainly gauge user behavior, not bugfixing the new tab page, but things like this diminishes its usability and all else equal would make users stick to old patterns.

    Reply

  5. Eduardo wrote on :

    Shouldn’t it be “higher” instead of “higer” on the title of the penultimate graphic?

    Reply

    1. Lilian Weng wrote on ::

      oh yes, fixed.

      Reply

  6. Trackback from If a website is set as the browser's home/start page, how many average visits will it get per day per user?... on ::

    If a website is set as the browser’s home/start page, how many average visits will it get per day per user?…

    A very large majority of Firefox users keep the default “Firefox Home” start page. In Firefox this page is actually locally hosted, a part of Firefox and not hosted on the Web, and we don’t track how many times a day it gets opened. We have an alter…

    Reply

  7. Pingback from Mozilla User Research – Test P… | BrettSinclair.net on ::

    [...] (Twitter) par Brett le 09-08-2011 Mozilla User Research – Test Pilot New Tab Study Results http://blog.mozilla.org/userresearch/2011/08/test-pilot-new-tab-study-results/ No [...]

    Reply

  8. zac spitzer wrote on ::

    I always remove the text from links on the bookmark toolbar,
    leaving just a strip of favicons, it vastly improves the usability,

    Reply

  9. A. Nonimous wrote on :

    Did you count the number of “new tab, typing google (in the url bar of course), clic on the first suggestion” ?

    Reply

    1. Jhonatan wrote on ::

      IE 7 (gasp!) is even worse. There is only one close btuton, and it is only on the active tab. It moves all over the place all the time and is very frustrating.

      Reply

  10. Salo wrote on ::

    A beiggr UI mistake in my opinion, was changing the secure lock icon the new colour and the fact that it isn’t displayed all the time in the same place at the bottom of the screen makes it a lot more difficult to spot.

    Reply

  11. SEEKDRIVERS.COM wrote on ::

    I enjoy what you guys are up too. This kind of clever work and coverage!
    Keep up the terrific works guys I’ve added you guys to my own blogroll.

    Reply

Post Your Comment