Test Pilot now offers a set of enhanced features to manage user studies. These features will be reflected in the Feedback channel if you are using Firefox 4 betas (so no need to download anything). If you have Firefox 3.6 or 3.5, you can upgrade to Test Pilot 1.1 here.
Last July, the Mozilla Labs team launched Test Pilot into Firefox 4 beta via the Feedback channel. Over the past eight month, with your support, we’ve rolled out several valuable studies and gained many insights that have helped improve Firefox. Meanwhile, our Test Pilot community is growing quickly – 3 million active users milestone! We want to continue delivering the best experience to the millions who help us improve Firefox, so over the past few weeks we’ve been working hard to update the extension with some important new features.
What’s New?
1.Targeted Study Deployment
Often times, we are interested in questions that apply to just a subset of our Test Pilot population; for example, the subset of Windows 7 users, or users with a particular non-default setting. With Targeted Study Deployment, we can now run studies on just the relevant subsets of users. Technically, these studies will be delivered to all Test Pilots, but will only run on profiles that match the pre-defined criteria, for example, computer operating system, Firefox version or preference settings.
2.Random Sample Deployment
Similar to Targeted Study Deployment, Random Sample Deployment gives us the ability to run studies on a random sample of our users. With over 3 million users, our studies can achieve statistical significance without covering the entire population. On the user side, Test Pilot will less frequently interrupt browsing with notifications, since users will no longer receive every study. (We are also working on a solution for those that would like it participate in all Test Pilot Studies.)
3. Stop Switch
This function offers the ability to turn off a study immediately in case there are any critical issues. Fortunately, we’ve yet to encounter any security issues, and we will continue to thoroughly review each study before its release. This Stop Switch feature provides an extra layer of protection on top of our review process.
4. Error Reporting
In the past we’ve relied on the Test Pilot discussion group to discover study or extension bugs. Now when you submit your study data to our server, it will also include a log of any errors that occurred. This will help us to debug and improve the study for the future.
5. UI Bug Fix
We now have also fixed a couple Test Pilot notification window bugs, including the transparent background bug that was discovered by several people. Thanks to Archaeopteryx [:aryx], Eitan Adler, Tim (fmdeveloper), Ryan Dunlop and Alexandr Lookoshkoff for filing the bug and sending us the screenshot!
What’s the next
1. Test Pilot for Mobile
Yes, we will work with Mozilla Mobile team to understand how Firefox on Mobile is used in the near future. It would also be fun to understand how Firefox are used differently between desktop and mobile devices. Try Firefox on your Android phone or Nokia N900 now and we are open to your feedback!
2. 2. Localization underway
Localizing each study for different languages is a goal for this year. We plan to solicit help from the localization and web development teams that localized our Mozilla web sites – if you want to join the effort, please let us know!