Test Pilot New Tab Study Results

[Cross-posted at Mozilla User Research]

The new tab page in Firefox is intentionally left blank, while some browsers present rich information on a newly opened tab.

The decision to leave new tab pages in Firefox blank was driven, in part, by a suspicion that too much information in the new tab may distract users from getting to the destination intended for the new tab. To test whether this suspicion is true and to learn more about user behavior after opening a new tab, Test Pilot recently released the New Tab Study and will soon release a multivariate test on the new tab page. Test Pilot is a platform collecting structured user feedback through Firefox. It currently has about 3 millions users and all the studies are opt in. You can help us better understand how people use their web browser and the Internet in order to build better products by participating studies. Test Pilot add-on is available here.The study ran for 5 days and in all, we collected 256,282 valid submissions.
Results of the study show that on average each user daily:
  • opens 11 new blank tabs
  • loads 7 pages
  • visits 2 unique domains
  • visits 2 pages in a new tab before they leave or close it

Below are details on how a user loads a page in a new tab, their intentions when opening a new tab, and time spent on new tabs below.

How do users load a page in new tabs?

 We detected 11 different methods to load a Web page in a blank tab page. Actions in the Url bar include pressing ENTER through keyboard, clicking the go button on the right side of the bar, clicking the Web page suggestions in the dropdown menu and pressing ENTER key for dropdown suggestions. Similarly 4 actions can be performed in the search bar too. Users can load a previously saved page from the bookmark bar in the toolbar or Bookmark/History in the menu bar.

Note:

  • The URL bar is most used when navigating to new websites.
  • The Search bar is also popular. Users rarely use search bar dropdown to look for old search terms.
  • The Bookmark toolbar is used more often than the bookmark menu button.
  • The History Menu button is seldom used.

We can also classify all methods for loading web pages into either keyboard-based or mouse-based category. Generally speaking, users have a slight preference for mouse usage.

 

Why do users open new tabs?

1.    Are they looking for a specific URL?

13.95% of new tabs (13,941,404) are opened while the text in the clipboard starts with “http” or “www”, which are very likely to be URL strings. The number is surprisingly high, although it may be caused by previous actions rather than by pasting for loading a specific URL.

2.    Users browse a limited set of domains, and only a small proportion of domains attract most visits

If we represent each user as a single point in the plot where x-axis is the number of pageloads, and y-axis is the number of unique domains visited, we can get the following graph. The dash line (diagonal) is what will happen if users always visit a different domain for each page load. When the users are not so active, pageloads less or around a few hundreds, the number of unique domains grows linearly. However, once users get to browse more, distinct domains tend to be stabilized and saturated.

Globally, we check the visit frequencies of all domains, and find that globally only 17.38% domains (461,133 unique domains in total) take 80% of the total page loads (8,291,541 pageloads in total). It verifies the famous “20-80” law of long tail phenomena.

On the individual level, we are interested in whether a single user performs the browsing movements according to the 20-80 law. For each individual, domains taking 80% of the total page visits is defined as “main domains”. A user can confirm the 20-80 law if the ratio of the number of his main domains to the number of distinct domains is around 20%. According to the following fig., active users browse more web pages everyday, but the number of primary sites they go to decreases proportionally. It suggests that when users visit more sites, they prefer to go to the same sites more frequently. The result supports the existence of a speed-dial new tab page to some extent.

 

Time Spent on New Tabs

According to the study results, on average, users open 2 pages in a new tab before they leave or close it. They load the first web page in 6 seconds (median) after they open new tabs, and stay on the tab for 1 minute (median) once they start browsing. The distributions of these two types of reaction timings display broad tails. The actually mean values are much higher than the medians: users load the first web page in 45 seconds (mean) after they open new tabs, and stay on the tab for 7 minute (mean) once they start browsing, since the outliners and expected noises can vary the mean value a lot.
Meanwhile, how users open a new tab can distinguish 2 groups of mouse-based users and keyboard-based users. The tabs invoked by “Plus Button” and “Double Click on TabBar” represent the group of mouse-based users, and the tabs invoked by “Command+T” represent the the group of keyboard-based users. The results turn out that keyboard-based users act slightly faster than mouse-based ones, and they can stay on the same new tab a bit longer.

The study is preliminary study for redesign requirement of the new tab pages in Firefox. We detect user behavior patterns of how they use the new tabs, including how they load a new page, broadness of domain visited, and the timing of different actions. In the following New Tab Multivariate Test, we will do a comparison between several designs of the new tab page, and more research questions will be answered, including whether too much information in the new tab may distract users from the original target or not.

9 responses

  1. Ed wrote on :

    “Test Pilot is a platform collecting structured user feedback through Firefox. It currently has more than 200 millions users”

    Do you mean more than 200 thousand?

  2. Simon wrote on :

    I’d note that Firefox’s default behaviour when using the search box is to open the results page in the current tab. I suspect you’ll find that’s a big factor in why people open a new blank tab – so they can search without losing their current page.

    Of course, the browser.search.openintab setting fixes that, but I don’t think many people know that it exists…

  3. Gerhard wrote on :

    The main reason I open a blank tab is to type in a keyword for a website I’ve already bookmarked that I access frequently.

    I assume that the use of keywords would be folded into the URL Bar Enter figures but it should possibly be separated out as it is an extremely useful feature that not many people know about but can save significant time.

  4. mucinch wrote on :

    I would love to use an interactive new tab page, but only if it offers equal or better performance than about:blank. If speed-dial takes more time than say clicking on a bookmark then it could be detrimental to my user experience.

  5. David Naylor wrote on :

    Very interesting to see the data. So I guess the conclusion is that it is very important not to distract keyboard users, while trying to help mouse users get more efficient.

    The idea of having different new-tab layout/look depending on keyboard/mouse that you (or someone else?) wrote about earlier is brilliant.

  6. Chris Barts wrote on :

    It spikes CPU usage on one core every time I change tabs. How do I get rid of it?

  7. Hank Roberts wrote on :

    So, when I open a new tab, I get a page with

    “Test Pilot New Tab Test
    Recover Removed Sites” and icons for pages I’ve opened sometime in the past.

    So far none of the icons have been pages I was intending to look at next. What’s the point here?

    Each when I hover the mouse has a star and an X.

    It seems to make no difference whether I click the star, click the X, or proceed as I intended to.

    What’s it doing?

  8. Gijs wrote on :

    I’m really surprised though. If I understand your post correctly, users in the study opened only 2 distinct domains a day?

    Right now I have 7 different domains open in this copy of Firefox (out of 9 tabs) and I literally find it hard to imagine to spend a day on only 2 domains, ever. So I must be misunderstanding what you mean, I’m just not sure how, yet…

    What happens if you exclude days where the user opened no domains at all / didn’t open their browser?

  9. Edli wrote on :

    The speed dial was the reason why I stuck with Opera for so long. It was so much more efficient than the blank, unused space of other browsers. In most of the cases I open a new tab to visit a bookmarked website so it makes sense to have bookmarks on that white space.

    Been using chrome since I installed the speed dial extension.