Reflecting on “It’s Our Research” at Mozilla

I thought we involved stakeholders in research pretty perfectly, here at Mozilla. Stakeholders come to research studies, listen attentively to research report-outs, and generally value the work our team does. Then I read “It’s Our Research”. I still think the Firefox User Research team has great buy-in across the organization, but after reading Tomer Sharon’s book, there are so many more things we could improve on! And what fun is a job, if there’s nothing to improve, right?

I’d like to call in some stakeholders to help me tell four short stories related to four pieces of advice Tomer Sharon provides in his book. (By the way, there are so many ideas in that book, it was hard to pick just four.)

This post is authored by Jennifer Davidson, with contributions from Julia Starkov, Jim Thomas, Marissa (Reese) Wood, and Michael Verdi

Let’s start with the “failures”. Failure is a big word, but they’re really two examples where I could’ve done better, as a user researcher.

 

Never skip a field debriefing

Tomer Sharon recommends that we never skip a debrief. A debrief includes those precious moments immediately after interviewing a participant or visiting a participants’ home. They capture your first reactions after the experience. Sharon details that they’re important because it helps prepare for analysis, and helps stakeholders remember the sessions. On the Firefox UX team, we have a great debrief culture. We reserve time after interviews, even if they’re remote interviews to capture initial thoughts. This practice is more formal when we do field research — where we not only have an individual debrief form that we fill out immediately after each visit, but we also do group debriefs together after each interview to talk together about what we observed. I would say we always do debriefs after every interview or session with our users, but then I’d be lying. Let’s have Julia Starkov, a Mozilla IT UX designer talk about her experience when we skipped a debrief in the field.

The time we skipped a debrief, as told by Julia: 

We wrapped up our Salem user research on a Friday afternoon; we were staying in different cities and some of us had plans right after the interview. We were also parked outside of the participant’s house so we decided to skip the group debrief and get the weekend started. While this felt like a huge relief at the time, I regret not pushing for us to meet at a cafe and wrap up the research together. By the time the weekend came and went, and I flew back to California, it was hard to recall specific parts of that interview, but mainly, I felt we could have definitely benefited from starting synthesis right away as a group. Overall, I thought the research effort was a total success, but I feel like I would have retained more insights and memories from the experience with a bit more team-time at the end.

 

Always include an executive summary

An executive summary is a few sentences, maybe one slide (if it’s in slide format) that summarizes a research study. I’ll tell you — it is the hardest part of a research study. Let’s have Marissa (Reese) Wood, Firefox VP of Product, explain the importance of an executive summary.

The importance of an executive summary, as explained by Reese:

The purpose of an executive summary is to describe the main and important points of your document in such a way that it will engage the readers. The intent is to have them want to learn more and continue to read the document. This is important because the reader will pay greater attention to detail and read the doc all the way through if they are engaged.  In short, without a good executive summary, the document will likely not get the attention it deserves.

Reese puts it nicely, but I’ll reiterate, without an executive summary, people may not pay any attention to the study results (a researcher’s worst nightmare!). So over the past year, the Firefox UR team has been iterating on our executive summary style, to better fit the needs of our current executives. We try to be more succinct with our executive summaries than before, with clear takeaways and calls to action for each report.

Now let’s move on to talk about a couple of successes.

Make it easy to participate

 

Be prepared — not only with great research practices like doing a pilot (a practice research session before doing the rest of the study, to work out the kinks in a research protocol) — but also, by making it easy for stakeholders to participate. This means that the researcher, or a supportive operations manager, needs to do some administrative tasks. When you have stakeholders come with you on home visits, get them the materials they need to take notes or photos. With every field research trip I do, I find little tweaks I can make to improve the stakeholder experience. Earlier this year, I took a little extra planning time, and with a great example from other team members (particularly, Gemma!), created a packet in a waterproof (that’s important if you’re doing research in the rainy Pacific Northwest like we were) document folder for each stakeholder. It felt like making personalized goodie bags for a birthday party. The packet contained everything they needed to participate: pen, sharpie, note-taking guide, photograph shot list, post-its, a legal pad, and a label with their name on the packet. And Jim Thomas, Firefox Product Manager, will talk about his experience participating in field research.

 

The time I participated in field research, as told by Jim:

As a Product Manager, I know how important it is to spend time connecting with your users, learning how they think about your product, observing what they do and how they react. I’ve done this informally throughout my career, but formal field research seemed like it would involve a lot more rigor. In fact, it was even more rigorous, but it didn’t feel like it thanks to the prep work put in by the team. Every session had our roles and responsibilities planned ahead of time, with relevant materials in a convenient package. Debriefing after each session was simple even at restaurants or picnic tables because everything we needed was at our fingertips. All the process was handled for me so I was able to focus on my most important goal: learning about our users and their perspectives.

 

Analyze together

 

Sharon recommends analyzing results together with stakeholders. He mentions that ideal outcomes from analyzing together are:

• “Stakeholders are more likely to act upon study results because they participated in the study’s development”
• “The design or product changes are more likely to be the right ones because they are based on multidisciplinary angles that try to solve the same problem”

Involving stakeholders in analysis can be tricky because analyzing data is difficult! It’s entirely too easy to use broad strokes to say what you think you saw, but to be rigorous with qualitative research analysis, we dive deep into the weeds with every action, every comment, to see if any themes or patterns emerge.

I mentioned earlier that Sharon recommends debriefing with stakeholders who are doing research with you. Notably, debriefs are not analysis. They may start the analysis juices going, but analysis is much more in-depth and complex.

We don’t always involve stakeholders in analysis, particularly if the timing is tight. However, in one case, I spent an extra few days in Germany, so we could do analysis immediately after field work. I think it went really well, and I was so happy to be about 80% done with analysis before even leaving Germany. Let’s hear what it was like for Michael Verdi, a Firefox UX designer and my partner on the project.

The time I participated in multiple days of analysis in Berlin, as told by Michael: 

I’m really happy I got to participate in the analysis phase. I’d done it on a previous project and found it extremely valuable. In addition, I thought we were fortunate this time to be able to begin right away while everything was fresh. It’s so good to spend time digging into details and carefully considering everything we saw. It’s also difficult (but great practice) to hold off on trying to solve all the problems right away. I feel like by the time I did move on to design I’d really absorbed the things we learned.

This is not the end

While this blog post was a great reflection exercise about my practice as a user researcher with involving stakeholders in research, it is not the end of the story. The practice of user research is just that — a practice that takes practice, and continuous improvement. Next year, I hope to work on other concrete ideas that our Firefox UR team can implement to increase stakeholder involvement in user research even more.

 

Thank you to Gemma Petrie and Elisabeth Klann for reviewing this blog post. 

Also published on medium.com.