Introducing Test Pilot

Chris Beard

47

Test Pilot is an idea for a new user testing program for Mozilla Labs that aims to build a 1% representative sample of the Firefox user base for soliciting wide participation and structured feedback for Labs experiments.Test Pilot

Presently, whenever Mozilla publishes a new experiment we rely upon the blogosphere and word-of-mouth to attract the attention of people interested in trying it out. We then hope that some number of those early adopters will come back and find the appropriate place to provide feedback.

The problem is that we are systemically biasing feedback towards only those who happen to hear about a particular experiment (on a given day) and in those cases skew to the technically-savvy early adopter. It is therefore very hard to derive conclusions representative of our much larger and increasingly mainstream user base.

We also have no structured way in which to request feedback, and in particular, positive feedback. People are much more likely to take the time to dig in and figure out how to provide feedback if they have a negative reaction. It’s also likely true that the less technical adopters are confused when faced with Mozilla’s panoply of feedback options.

And finally, at the end of this “testing process,” we effectively throw away the interested people who have taken the time to volunteer as testers and start over again from scratch the next time around.

Overview of Test Pilot
We can provide a much more satisfying experience all around by putting in place some basic infrastructure. Here’s the idea:

  • We develop and promote a formal Test Pilot program with a Firefox add-on at its core.
  • The first time the Test Pilot add-on is run, it asks a few simple non-personally-identifiable questions in order to put the user into a demographic bucket, e.g. technical level, locale, etc., and to let them opt in to additional anonymous instrumentation.
  • Test Pilot will then notify its users when a new experiment is available for testing. If the user opts in, it will download the required software (if any) and load any information required to get started with the new experiment, e.g. overview, use cases, etc.
  • After either a specified amount of time or upon completion of a specific action, Test Pilot will prompt the user for feedback. The feedback form will only ask a few questions selected from a much larger set. A link will be provided to provided more comprehensive unstructured feedback or bug reports.
  • The set of questions posed for feedback will be randomly distributed within each demographic bucket to ensure statistical significance of the results.
  • Anonymized aggregate results and analysis will then be posted automatically to the Test Pilot site.
  • All participants will receive a “flight badge” displayed in their Test Pilot profile and available to embed on blogs, social networks, etc.

The idea is that by reducing the amount of required feedback to only a few clicks we can increase overall rates of participation.

While this will not take the place of our traditional testing and feedback mechanisms, Test Pilot should provide a new set of tools and capabilities to increase the efficiency and effectiveness of our user testing. It should also allow us to grow our community and mindshare with people who want to help but to whom we haven’t yet given the opportunity to participate.

Roadmap
We’re pulling together a basic framework now with plans to launch the first version within the next few weeks. Specific features and roadmap are being developed in the Test Pilot discussion forum on the Mozilla Labs site.

Getting Involved

  • Discuss, debate, and add to the design in the Labs discussion forum
  • Jump into #labs on irc.mozilla.org

47 responses

  1. Tiffanie wrote on :

    Would love to test also. Sign me up too.

  2. Mike O wrote on :

    I would also like to be a part of this. Please sign me up as well.

  3. Ringthane wrote on :

    PLZ sign me up. I would really like to be a part of this.

  4. Dawn Albano wrote on :

    Sign me up too, I would love to help.

  5. Hamit Giray Nart wrote on :

    Count me in. I also can do Turkish localization. Let me know.

  6. Roger W wrote on ::

    I’m interested I’m a novice can’t type well or fast, I’d still like to help

  7. Lord_Possum wrote on :

    This is a GREAT Idea. I hope it will help for develop a powerful Firefox!
    I want to be a test pilot!
    Sorry for the english!

  8. RBango wrote on :

    I’d like to get in on this, if possible. Let me know.

  9. Jedi Esparza wrote on ::

    I Can Haz Test Pilot for Firefox?

    Seriously, would definitely like to know more on how to sign up.

  10. Cat Peterson wrote on ::

    Happy to help. Let me know when you want folks to volunteer! Cat

  11. Ron wrote on :

    Sign me up too, would love to help if I can.

  12. paco wrote on :

    no entiendo lo que quiere decir Website ssique no lo puedo rellenar solo pido que de Mocilla Firefox no tenga que sali por nada del mundo me fio de este sistema tampoco quiero perder mi correo que es portiz47@gmail.com muchas gracias por dejarme escrivir lo que pienso

  13. fuzzyclick wrote on :

    hello world. testing presently extensions and script to improve the browsing confort & ergonomy, i’d like to be able to give you some feedback on the general use of firefox. how can i subscribe to the “test pilot” program ?

  14. Vinicius Celms De Augustinis wrote on :

    Eu quero muito compor este time de contribuidores para o Firefox pois tenho uma grande conhecimento de Internet e trabalho com a área de suporte técnico e também sou uma defendedor e um protetor do Firefox, sendo ele o meu preferido para o uso.

    Este projeto para mim é de suma importância e de grande avalia para os seus usários. Mais uma vez eu tenho de parabelizalos por mais esta iniciativa e contem comigo para a melhoria deste software.

  15. Yann Stefan wrote on :

    Please let me know how to subscribe.

  16. Niklas wrote on ::

    I’d love to be a part of the testing process. I’m running Firefox on Windows and OS X, mostly the 3.1 beta 2-version. Please feel free to e-mail me regarding this if you please. Thanks!

  17. Kenny Villacorta wrote on ::

    How do I sign up or where do we download? Keep up the great work!

  18. Eric Reed wrote on :

    Please sign me up.

  19. Guillaume wrote on :

    Cool !
    Sign me in. keep me posted when DL is available

  20. Sadiq wrote on ::

    That would be nice to see in action.. Good luck guys.. I’l be in the first ten person to dl this :)

  21. Ben wrote on ::

    Great idea, can’t wait to take part in it!

    You know what another would another great idea? If you removed all pings from the comments and moved them to a separate (collapsed) location on the page, like most good blogs do.

  22. Zim wrote on ::

    Sounds like a nice project.

  23. neko wrote on :

    I’d love to participate, even if i’m one of your horrible early adopters ;D

    you’ll be getting at least 2 test cases out of me, one mac and one XP with portablefirefox

  24. David Busby wrote on :

    I would like to participate as long as we can see some of the results I often run into issues and end up trying to find out if it is Firefox, windows, or some combination thereof. Is this also going to be something we can install into fennic?

  25. wolffkrieger wrote on :

    ready to serve my browser! going to dl as soon as download link is available… xD…

  26. alex006 wrote on ::

    The image of the Firefox Test Pilot, on the top of the post, is available in a bigger size or in vectorial version ? It’s Free, Open Source, Creative Commons or closed ?

    Thanks!

  27. Pingback from The PLACES I’ve Been and the PLACES I’ll Go. on ::

    […] data collection, with the simple logging tool Spectator returning to compatibility recently and the Mozilla Labs participant pool (aka Test Pilot) […]

  28. Pingback from Business News Research » Mozilla CEO John Lilly Reveals More Details On Stealth Data Project on ::

    […] In a blog post, he says “data is one of the most important pieces to faciliate understanding (and innovation), and is also one of the most under-explored areas of the modern web.” He also says that Mozilla has two early projects that touch on the idea – Spectator and Test Pilot. […]

  29. Pingback from Mozilla CEO John Lilly Reveals More Details On Stealth Data Project on ::

    […] In a blog post, he says “data is one of the most important pieces to faciliate understanding (and innovation), and is also one of the most under-explored areas of the modern web.” He also says that Mozilla has two early projects that touch on the idea – Spectator and Test Pilot. […]

  30. Pingback from Aufregung um geheimes Daten-Sammel-Projekt von Mozilla » Daten, Firefox, Mozilla, Lilly, Namen, Mozilla-Chef, Stadium, Nutzer » TheXBlog on ::

    […] helfen soll, das User-Interface des Browsers zu verbessern. Das andere hat den Namen Test Pilot und soll eine repräsentative Gruppe von Firefox-Nutzern zusammenbringen, die den Entwicklern […]

  31. Pingback from John’s Blog » Blog Archive » Mozilla, Firefox & Data on ::

    […] used to improve the user interface of Firefox, and another is a project in Mozilla Labs called Test Pilot. They’re both early and very limited in […]

  32. Pingback from Lancement de Test Pilot | FirefoxBlog on ::

    […] Test Pilot (pilote d’essai, en français) est une extension destinée à permettre aux utilisateurs de Firefox 3 de donner leur avis sur différentes fonctionnalités du logiciel. […]

  33. Natanael L wrote on :

    I’ll participate in this when it’s ready!

    I have Ubuntu 7.10 (will install 8.04 when out of beta) and MS Vista Home Premium and are using both FF2 and FF3b5.

  34. Seattle John wrote on :

    I would love to participate. I have plenty of time to play on ‘this new fangled machine’. Currently running FF 3.0 B5 with Weave 0.1.xx on MS Vista Home Prem.
    Let me know ASAP it is ready to dl.

  35. Pingback from ¿Cómo conseguir retroalimentación por Internet para tu empresa? « Skylab on ::

    […] gente de Mozilla Labs está perfilando una idea, bajo el nombre de Test Pilot, que tiene como fin obtener retroalimentación -selectiva- de una muestra representativa del 1% de […]

  36. David Naylor wrote on ::

    Sounds like a great idea!

    But surely, if you don’t somehow include it randomly in the Firefox installs (like talkback used to be included) then you will still only be getting geeks involved?

  37. Pingback from l word on ::

    […] site of the National Civil Rights Museum. Todahttp://www.centredaily.com/business/story/496981.htmlMozilla wants unbiased testing reports, creates Test Pilot&quotTest Pilot is an idea for a new user testing program for Mozilla Labs that aims to build a 1% […]

  38. Pingback from Mozilla Labs » Blog Archive » Introducing Test Pilot at TweetLinked on ::

    […] Blog Archive » Introducing Test Pilot (reference) (more) I want to be a Mozilla Test Pilot! labs.mozilla.com/2008/03/introducing-test-pilot/ Filed under Uncategorized […]

  39. Pingback from March 31st, 2008 23:53 GMT at TweetLinked on ::

    […] Blog Archive » Introducing Test Pilot (reference) (more) I want to be a Mozilla Test Pilot! labs.mozilla.com/2008/03/introducing-test-pilot/ Filed under Uncategorized […]

  40. Pingback from March 31st, 2008 23:53GMT at TweetLinked on ::

    […] Blog Archive » Introducing Test Pilot (reference) (more) I want to be a Mozilla Test Pilot! labs.mozilla.com/2008/03/introducing-test-pilot/ Filed under Uncategorized […]

  41. Pingback from March 31st, 2008 23:53:22 -0400 at TweetLinked on ::

    […] Blog Archive » Introducing Test Pilot (reference) (more) I want to be a Mozilla Test Pilot! labs.mozilla.com/2008/03/introducing-test-pilot/ Filed under Uncategorized […]

  42. Kurt (supernova_00) wrote on :

    Introducing? Well where is the download link?

  43. Pingback from March 31st, 2008 23:53:22 at TweetLinked on ::

    […] (reference) (more) I want to be a Mozilla Test Pilot! http://labs.mozilla.com/2008/03/introducing-test-p… Filed under Uncategorized […]

  44. Pingback from 1207022002 at TweetLinked on ::

    […] (reference) (more) I want to be a Mozilla Test Pilot! http://labs.mozilla.com/2008/03/introducing-test-p… Filed under Uncategorized […]

  45. Joey wrote on :

    Love the idea,
    Would defenitely participate when the testpilot plugin is available.
    Keep up the good work!

  46. lpalli wrote on ::

    The image of the Firefox Test Pilot, on the top of the post, is available in a bigger size or in vectorial version ? It’s Free, Open Source, Creative Commons or closed ?

    Thanks!

    Luca

  47. Trackback from foxiewire.com on ::

    Introducing Test Pilot…

    Test Pilot is an idea for a new user testing program for Mozilla Labs that aims to build a 1% representative sample of the Firefox user base for soliciting wide participation and structured feedback for Labs experiments….