Firefox Test Pilot is an approach to experimenting with new browser features in the open. What this means is that we plan, design, build, and evaluate experimental features with as much transparency as possible, in line with Mozilla’s mission and values. In this second year of Test Pilot, we’re making a concerted effort to make our work even more public by sharing all of the user research we conduct on our experiments.
What we already do in the open
From the beginning, Test Pilot has strived to work in the open. For example, the majority of our meetings are open to the public. We publish our documentation in places like the Mozilla Wiki and Github. The latter has also been a place where people trying our experiments can suggest and vote on enhancements.
The Test Pilot team prioritizes enhancements to our Containers experiment based on up-votes on Github.
We also receive feedback on experiments via Twitter and the Discourse forum. For example, people using our Tab Center experiment reached out to us on Twitter when they heard that the experiment would no longer be supported starting with Firefox 56.
Whenever an experiment graduates, we publish — for anyone to see — what we learned from that experiment, including the metrics that informed those learnings.
Detail from our graduation report on the No More 404s experiment
Transparency beyond code
Even with these efforts, we realize that, historically, transparency in open source development has largely focused on code and the conversations that happen around that code. What that focus has neglected are other important parts of our process for creating the products we ship, including user research.
Test Pilot user research has not been shared widely in the past because of the extensive personally identifiable information (PII) contained in our qualitative research. In most cases, we have not removed PII from our research findings because, for the Test Pilot team, those details are often critical for helping us build greater empathy for people using our experiments and for internalizing insights from the research we conduct. For example, a grimace and silence captured on video from a research participant trying to use one of our experiments is generally much more affecting and memorable than a written, anonymized quote from a research participant about how an experiment was confusing.
The benefits of making user research more open
We will continue to leverage user research containing PII internally when the Test Pilot team can learn from it. However, we realize that there are benefits that we can reap by making our user research more shareable beyond our team. With a new commitment to sharing our user research in the form of blog posts here on Medium, links to our research reports, and embedding insights from user research in public conversations happening in other channels like Github, we aim to:
- Possibly expand who can contribute to our work to include people who will share ideas and feedback on our research approaches
- Increase accountability around how we leverage user research findings to inform the design and development of our experiments
Share ideas about open research
We welcome additional ideas about how we can make our user research more open while maintaining its rigor and integrity. If there are open research initiatives that you think we can learn from, let us know. We look forward to hearing your thoughts on the research we share moving forward.
Originally published on medium.com