As announced last year, we’ve been slowly working on a new project to provide an opt-in framework for collecting and sharing Web usage information in the interest of improving the online experience. The first priority though, will be sorting out how best to ask people to participate, collect data appropriately, and share that data and any analysis — while ensuring that individual privacy and security is preserved.

How many tabs does an average user use at a time? How about novice users? How often is the stop button pressed? How many times do people open a new tab to perform a search?

There are hundreds of questions like these whose answers would help quantitatively inform the design process of Firefox. At the moment, as evidence in discussions we generally only have access to studies, anecdotes, first principles, early-adopter feedback, and ad-hoc experiments.

While those are all useful, they are no substitute for specific data.

Enter Test Pilot. It’s a still-in-concept platform for a new user-testing program for Mozilla Labs that aims to build a 1% representative sample of the Firefox user base for soliciting wide participation and structured feedback for interface and product experiments.

Scalable

Its not just Firefox that needs a usability lab. Thunderbird needs one. Seamonkey needs one. Every Mozilla Labs project needs one. Test Pilot is a platform — starting as a Firefox extension — on top of which anything can be put through usability-testing boot-camp.

As Labs scales to having hundreds of projects, we’ll need a way for any researcher to ask usability questions, and get meaningful answers back quickly.

Building a usability lab for millions of testers is new territory. Is it possible to use the built-in camera on laptop’s to do rudimentary gaze-tracking studies? We’ll find out.

Test Pilot is, simply put, part of the massively scaled open usability lab we’re aiming to build at Mozilla Labs.

How It’ll Work

The first time the Test Pilot add-on is run, it will ask a few simple non-personally-identifiable questions to put you into a demographic bucket, e.g. technical level, locale, etc., and to let you opt-in to additional anonymous instrumentation.

The idea so far is that we’ll only keep and publish aggregate data and only under open-content licenses. We’d also have some process by which we’d review every test to make sure your privacy is held sacred. Once in a while you may be asked to participate in a short survey based on your demographic. If you’ve opted into allowing additional anonymous instrumentation, an experiment may request some of that information for aggregated study.

That’s it. You’ll have become part of the global Mozilla community by participating and providing your input into the open design process.

On Demand Questions

Unlike traditional usability instrumentation, Test Pilot won’t constantly record data — instead it will only record the data necessary to answer a question posed by a researcher (and that you’ve opted into). Whether it be a survey or click-stream, a minimum amount of privacy-protecting data is sent back for analysis.

Participating in the testing process should be optional and as simple and unobtrusive as possible.

Open Research

Mozilla follows an open and transparent development process. This also holds for usability data and we’d hope that all aggregate anonymous data could be published for anyone to access and research. In the same way, anyone would be allowed to create research tests and — pending review — have them run. Test Pilot is useful for both developers trying to prioritize features and academic researchers performing in-depth studies.

In conjuntion with projects like IBM’s Many Eyes, Test Pilot becomes a rich source of exploration for previously unaccessible data.

Secure and Private

To be clear, security and privacy is of the utmost importance when dealing with user data. Just like Firefox, Test Pilot will always honor your privacy and would be completely opt-in. Again, data would only be reported in anonymous aggregate forms and any and all personally identifiable would be stripped out.

Join Us

There are many ways you can get involved. Come brainstorm at a Lab’s Night, participate in the forums, hop on IRC, or help create on our wiki.

We’re also hiring: If you are interested in helping to formulate, plan and build Test Pilot in a full-time capacity, it’s time to break out the CV and apply. We’re looking for an awesome project lead who can think big-picture and execute on the details. Apply here.

1/23/09 – Updated to clarify that Test Pilot will be a completely opt-in program and the idea is that all data would be kept and reported only in anonymous aggregate formats. We’re still working on the idea, and would love your feedback on how we can do this right.