Test Pilot Update

As announced last year, we’ve been slowly working on a new project to provide an opt-in framework for collecting and sharing Web usage information in the interest of improving the online experience. The first priority though, will be sorting out how best to ask people to participate, collect data appropriately, and share that data and any analysis — while ensuring that individual privacy and security is preserved.

How many tabs does an average user use at a time? How about novice users? How often is the stop button pressed? How many times do people open a new tab to perform a search?

There are hundreds of questions like these whose answers would help quantitatively inform the design process of Firefox. At the moment, as evidence in discussions we generally only have access to studies, anecdotes, first principles, early-adopter feedback, and ad-hoc experiments.

While those are all useful, they are no substitute for specific data.

Enter Test Pilot. It’s a still-in-concept platform for a new user-testing program for Mozilla Labs that aims to build a 1% representative sample of the Firefox user base for soliciting wide participation and structured feedback for interface and product experiments.

Scalable

Its not just Firefox that needs a usability lab. Thunderbird needs one. Seamonkey needs one. Every Mozilla Labs project needs one. Test Pilot is a platform — starting as a Firefox extension — on top of which anything can be put through usability-testing boot-camp.

As Labs scales to having hundreds of projects, we’ll need a way for any researcher to ask usability questions, and get meaningful answers back quickly.

Building a usability lab for millions of testers is new territory. Is it possible to use the built-in camera on laptop’s to do rudimentary gaze-tracking studies? We’ll find out.

Test Pilot is, simply put, part of the massively scaled open usability lab we’re aiming to build at Mozilla Labs.

How It’ll Work

The first time the Test Pilot add-on is run, it will ask a few simple non-personally-identifiable questions to put you into a demographic bucket, e.g. technical level, locale, etc., and to let you opt-in to additional anonymous instrumentation.

The idea so far is that we’ll only keep and publish aggregate data and only under open-content licenses. We’d also have some process by which we’d review every test to make sure your privacy is held sacred. Once in a while you may be asked to participate in a short survey based on your demographic. If you’ve opted into allowing additional anonymous instrumentation, an experiment may request some of that information for aggregated study.

That’s it. You’ll have become part of the global Mozilla community by participating and providing your input into the open design process.

On Demand Questions

Unlike traditional usability instrumentation, Test Pilot won’t constantly record data instead it will only record the data necessary to answer a question posed by a researcher (and that you’ve opted into). Whether it be a survey or click-stream, a minimum amount of privacy-protecting data is sent back for analysis.

Participating in the testing process should be optional and as simple and unobtrusive as possible.

Open Research

Mozilla follows an open and transparent development process. This also holds for usability data and we’d hope that all aggregate anonymous data could be published for anyone to access and research. In the same way, anyone would be allowed to create research tests and — pending review — have them run. Test Pilot is useful for both developers trying to prioritize features and academic researchers performing in-depth studies.

In conjuntion with projects like IBM’s Many Eyes, Test Pilot becomes a rich source of exploration for previously unaccessible data.

Secure and Private

To be clear, security and privacy is of the utmost importance when dealing with user data. Just like Firefox, Test Pilot will always honor your privacy and would be completely opt-in. Again, data would only be reported in anonymous aggregate forms and any and all personally identifiable would be stripped out.

Join Us

There are many ways you can get involved. Come brainstorm at a Lab’s Night, participate in the forums, hop on IRC, or help create on our wiki.

We’re also hiring: If you are interested in helping to formulate, plan and build Test Pilot in a full-time capacity, it’s time to break out the CV and apply. We’re looking for an awesome project lead who can think big-picture and execute on the details. Apply here.

1/23/09 – Updated to clarify that Test Pilot will be a completely opt-in program and the idea is that all data would be kept and reported only in anonymous aggregate formats. We’re still working on the idea, and would love your feedback on how we can do this right.

32 responses

  1. Alex Brookmyre wrote on :

    Really interested in this, not a programmer or anything like that but I know my way around computers and will be able to report any bugs I find. 😀

  2. Alex Brookmyre wrote on :

    Really interested in this, not a programmer or anything like that but I know my way around computers and will be able to report any bugs I find. 😀

  3. mike wrote on :

    What a smart idea.

  4. Jerry wrote on :

    Outstanding concept, looking for true feedback from users, what will they think if next? 99% of today’s companies would shrink back in horror at this concept….

  5. gerdk wrote on :

    Hello,

    I would like it very much to participate in this project.
    I’m looking forward to some activities….

  6. Stanley Baker wrote on :

    I am not a programmer or expert but I am willing to help if it will be of assistance. I have done beta work in the past and know my way around computers so if that is of any use please include me and I will be happy to make a difference..

  7. eva wrote on :

    I tried to install this, but the button does nothing.

  8. John Lenihan wrote on :

    Sorry but im not buying it.
    Why don’t you people just start
    a tech support site…instead of
    trying to sell personal info
    to other companys for a profit.
    You people are getting dumber by the day.

  9. Jeff Hubert wrote on :

    I’m in. When and where do we go?lol
    Juat keep searchin’ add-ons?

  10. TsarNikky wrote on :

    This is an excellent idea! As part of your demographic data collecting, perhaps you may want to supply some kind of descriptive narrative for each labeled level of expertise. Typical labels, such as “Novice, Intermediate, Advanced, Techno-wizard, Professional” meaningless by themselves.
    Also, be specific as to just what kind of data or activities are being tracked. Are you tracking URLs visited, or just different URLs? (Some people may not mind you tracking the fact they visit the same 3-4 porn sites a day, but do mind that you are capturing the URL.)

  11. danish wrote on :

    count me in

  12. Adrien Joly wrote on :

    Hi, in the frame of my PhD thesis, i’m currently writing a firefox extension that watches currently opened tabs in order to determine the current subject of focus of the user. I would be happy to work together if we can find some convergence.

  13. Lorenzo Pasquinelli wrote on :

    I’m also interested in participating in this somehow.

  14. William wrote on :

    Contact me if you need some people to 6test ill be willing to help test!

  15. Lars wrote on :

    Due to the various attacks of the government on privacy in the US and in germany, I believe that Test Pilot will have to fight a long way. However, due to the open-source nature, it has a much better stand than the “anonymous usage report”-features of other proprietary products.

    What I think is a must for the tool is a secure transmission of the data over https.

    Also nice would from the developers perspective could be an option, if the user wants to be contacted by the Mozilla Dev Team (as opt-in of course). This would allow to filter all results and e.g. ask this groups of users to participate in an additional survey (e.g. to clarify, why users did this or that).

    I’d also like to know, if the database will be publically accessable to other researchers (minus the personal data of users, who chose to allow contact from the Mozilla Dev Team). This would especially help students to get some interesting statistical data, how to analyze it and what appropriate conclusions may be drawn from the data. This could finally lead to a better education in the usability sector, which is still pretty weakly teached in teaching facilities.

    Regards!

  16. Edward Steckis wrote on :

    You should at least start creating a signup mailing list so I can sign up as a volunteer for test pilot and receive an email with a link so I will know when it is ready to download and install.

    Thank You Firefox 🙂

  17. wilson wrote on :

    Hopefully capturing of stats will awaken the FF team about how often FF consumes over 1GB of RAM, and eventually crashes. I really enjoy having the same browser across the OS platforms, but this memory problem is one of the reasons many of my friends have moved to alternative solutions.

    I’d be very interested in contributing my browser usage stats to the project.

  18. Tomas – University Place, WA wrote on :

    Very interesting. Seems to me that one of the most difficult problems presented is how to get ‘volunteers’ outside the normal range – those folks who are almost always slipping in under the radar.

    In any case, if you need to fill the “old, retired, handicapped, male, MacBook, Firefox” demographic, I’m available. 😛

    Take care,
    Tom

  19. Luke wrote on :

    Release date? also email updates when new releases and plans for the test such how long the test will run for and also what the data is being used for.

  20. Guitou wrote on :

    Great idea.
    Sign me in.

  21. Tom Allan wrote on :

    Great Idea!

    Keen to know when its out,
    anything to improve the best browser ever.

  22. Miles Prower wrote on :

    Willing to take part.

  23. Anonymous wrote on :

    Hi, The refresh button…i think this button is not used much, you think?

  24. Omnisilver wrote on :

    What about the data : will the user get an ID number which will be send with each data, or will they be sent by Test Pilot just with the user’si demographic bucket data ?

    Thanks

  25. Lisandro Pardo wrote on :

    Any idea when the add-on will be available?

  26. meoft wrote on :

    Hi
    I use actively firefox with 310 bookmarks !
    I want be a part of this survey.
    Thanks.

  27. jerry tommet wrote on :

    no comment, firefox is the best

  28. ChevignoN wrote on :

    Ok when you want =)

  29. T. Hebert wrote on :

    Hi, Is Test Pilot still alive and moving forward? It’s most definitely a needed development component.

  30. james herbert wrote on :

    novice user for test pilot,i recently found myself out of work,to make productive use of some of my time i decided to teach myself the ins & outs of the computer.at 42 i guess i am a little slower learning wise than i was when i was younger,it’s been a battle from day 1, so if you need novices & my mishaps & blunders can be of any use on projects, contact me, the novice.

  31. paul sorenson wrote on :

    I’m also interested in participating in this somehow – I am a human factors engineer with ~23 years experience in the high tech and consumer elec. industries

    pfs

  32. paul sorenson wrote on :

    this is a fascinating idea – one that IBM has been pursuing for the past few years with internal(proprietary) web-based data collection – if they are willing to discuss – you might learn a bunch from their experience.

    Let me know if you are interested, and i’ll introduce you to a few Sr. folks there so you can begin a dialogue.

    best regards
    pfs
    Human Experience Lab, Inc
    4888 NW Bethany Blvd
    Suite K5, #262
    Portland, OR 97229
    (c/w) 503.593.2745
    (F) 503.614.9554
    pfsorenson@verizon.net