Test Pilot Program Reaches One Million Active Users

Jinghua Zhang

It was just four months ago when we reported that 170,000 Firefox users joined Test Pilot, our open user research platform. Today, we are happy to announce that more than 1 million Firefox users are active Test Pilots, and the community continues to grow.

The Test Pilot program launched in August 2009 as a Firefox Add-on so users can help Mozilla understand how people use Firefox. as we designed in the privacy policy, the usage data is collected through pre-defined studies and participants can choose to submit their data or not after a study ends.

All Test Pilot studies, resulting insights and aggregated data are shared under Creative Commons licenses for the benefit of the Open Source community. The results have triggered some interesting discussions within the wider community. For example, people did their own analysis of the tab data from our first study as well as use that data to assist other research topics.

When Firefox 4 Beta was released In July 2010, a version of Test Pilot called “Feedback” was included in order to better understand how the browser is used by beta testers. Firefox 4 Beta testers don’t have to download the Test Pilot add-on and still have the choice to submit their data or opt out from a particular study as well as the whole program. Being included in Firefox 4 Beta helped Test Pilot recently cross the 1 million user milestone.

In the last year, we released 12 studies including log studies and surveys. The most recent study is a Firefox 4 Beta UI study, where we learned how people use interface elements through an interactive heatmap; this heatmap also allows us to compare usage patterns with a previously created Firefox 3.6 heatmap.

We couldn’t gain all this knowledge without the help of our Test Pilot participants. They not only contribute their usage data via focused studies, and also share unique use cases to help us design studies better, provide new study ideas, give corresponding UI suggestions, report bugs, and even fix programming syntax errors. We really appreciate all your help!

Moving forward

We accumulate tremendous data sets with Test Pilot that are valuable to the entire Open Source community. In order to share this knowledge with the community, we released the first Mozilla Open Data Analysis Competition. The goal is to encourage people to make sense of data. Please let us know (via email: jinghua[at]mozilla[dot]com) if you are interested to collaborate with us on it.

Get Involved

Besides calling participants for the Data Analysis Competition, we are also looking for people to help us design and implement user studies. If you have an interesting study idea or want to try other research methods on the Test Pilot platform, please submit your proposal or let us know in the discussion forum.