Summary: In a survey of 966 visitors to the Update page, 47.8% open new empty tabs from the new tab button, 30.4% from the file menu, and 21.8% from a keyboard shortcut.
In addition to running A/B and multivariate tests, we have begun supplementing our quantitative analysis with more qualitative feedback. Historically, many of our most impactful insights have come from survey data.
With this in mind, we recently launched a usability survey on the Firefox Update page page. Our goal? Understand how users open new empty tabs.
The Update page is an ideal place to run this survey because nearly every Firefox user views the page once. Additionally, because this page receives such high traffic, we are able to expose the survey to ~1% of users and get statistically significant results within a week.
Of the 6000 visitors who saw the survey, 966 responded–resulting in response rate of 16.1%. The data breaks down as follows:
Our early results are encouraging, but we can make a number of improvements. First, we may reduce our response bias by randomizing the order in which the responses are presented. Second, we will have more representative data if we run the survey immediately after a dot release. This will be a factor if the population of users who update Firefox immediately and the population of users who update later are heterogeneous (a likely assumption). Third, if we collect a larger sample, we can segment our analysis by operating system. We can run this analysis now (using SiteSpect’s excellent web optimization tool), but our results won’t be statistically significant. And finally, we can elicit a higher response rate by testing a few different survey styles.
Update page surveys provide one additional benefit. By comparing our survey results with our Test Pilot data, we can roughly approximate the bias in the Test Pilot population. Both data sets are biased, but our survey data should be more representative because it has a lower barrier to entry and a much larger sample.
Next up? We want to determine what’s driving the unusual traffic on the First Run page. Ken already has a survey in the works.
Have a survey question that would help us build a better Firefox experience? Please leave it in comments!