Meet Test Pilot in Firefox 4 Beta

Jinghua Zhang

We are excited to announce that Test Pilot will closely support the soon to be released Firefox 4 beta, playing an important role in the development of the next generation of Firefox through a new Add-On called “Feedback.” With this bundled Add-On, beta testers will be asked to participate in user interface studies, and that data will be collected and analyzed by the Test Pilot program.

Since we rolled out the first Test Pilot study last September, many participants have joined the Test Pilot program. With the help of the broader community, participation has grown from 5,000 to over 12,000 submissions to date, completing six integral studies. The Account and Password study, the Tab Switch study, the Menu Item Usage study and the Main Window Usage study have all provided important insights for the Firefox UX and product teams. With Firefox 4 on its way, the Test Pilot team will monitor, analyze and work to better understand the variety of user behaviors and needs resulting from the new user interface and features throughout the complete Firefox 4 beta development cycle.

The Feedback Add-On

Firefox 4 Beta will include an Add-On called “Feedback”, which is a channel that allows users to input free-form comments and join the structured user studies as part of the Test Pilot program. This Feedback Add-On is being customized for use in Firefox 4 Beta – and future betas. With the “Feedback” Add-On (see the tentative UI below) in the yet to be released Firefox 4 Beta, everyone can contribute and make Firefox better.

As always, security and privacy are priorities when Mozilla is dealing with user data. All privacy settings that the Test Pilot program has developed to give users control over their data will remain the same in the Feedback Add-On. These include:

  • Participants’ data will be transmitted to Mozilla only when they take all of the following actions:
    * Join the Firefox 4 Beta program by downloading the beta with the Feedback Add-On.
    * Submit data when the test is finished. Participants will be able to review all the data before choosing whether or not to submit it.
  • Test data will be stored anonymously and in aggregate. None of it will be publicly associated with any personally identifiable information about the participant.
  • Participants can quit a Test Pilot study before they submit any test data.
  • Participants can opt-out from all user studies or disable the entire Feedback Add-On at any time. Learn more.

How It Will Work

For people who are unfamiliar with the Test Pilot program,this is how Test Pilot studies will work in the Feedback Add-On for Firefox 4 Beta:

  • The first time the Feedback Add-On is run, the user is asked to complete a short, non-personally-identifiable survey in order to better understand the user from a demographic point of view, e.g. technical level, locale, etc.
  • The Feedback Add-On will then notify participants before a new study starts. Users will be able to view the detailed study description, e.g. the study goal, what data will be collected, how long the study will last, etc. Users can then choose to opt-out from this study.
  • After either a specified amount of time or upon completion of a specific action, the Feedback Add-On will prompt the user for submission. Users will be able to see their test data in a graphical format before they submit it to the Mozilla servers. User can also save and export the test data if they are interested in viewing the raw data.
  • All participants will receive a “flight badge” displayed in their Test Pilot profile after they contribute to a study.

Open Research

As with previous Test Pilot studies, all new Firefox 4 Beta studies will be open and conducted with the user’s permission. Here are a some of the questions that we are interested to learn more about in Firefox 4 Beta:

  • How do people initiate a new tab? Through the URL bar, history, bookmark or other actions?
  • How do people manage their browser preferences?

Data from all studies will be aggregated and made available under the Creative Commons Attribution 3.0 United States License to the public. In other words, the only information made public will be statistical information about the Test Pilot community as a whole. No data about any individual user will ever be made public. We encourage people to make creative use of this data, to conduct analysis and to perform research on these download-able data sets.

We are also open for receiving your own study proposals. Please suggest any study topics you’d like to see through the discussion group.

Stay tuned for the coming Firefox 4 Beta and the new Feedback Add-On!