Martin Best Blog

It has been almost 6 months since the project started so I thought it was way overdue to recap how things have been going.  I am excited about the interest this project is receiving from both active Mozilla Contributors (employees and volunteers) and the academic community.  I’m working hard to deliver on your hopes for direct access to the results of this project.

Update on the Interviews

The Bugzilla Anthropology project has been moving along well when it comes to the interview work.  We have completed 20 interviews plus 2 write-ins during the first quarter of this year.  The raw transcripts are posted at the following link:

https://wiki.mozilla.org/Bugzilla_Anthropology#Interviews

Work has been ongoing to parse through and summarize the results of these interviews into a digestible form.  We have had considerable help from Olga Baysal and Reid Holmes from the Cheriton School of Computer Science at the University of Waterloo who are working tirelessly to parse through all the data we collected.  They are categorizing the 1,213 comments gathered over the course of the interviews.  This will result in two documents, the first is a 3 page summary that available at the following link:

https://wiki.mozilla.org/images/4/4a/BA_-_3_page_summary.pdf

This initial document discusses how the final longer summary will be categorized and how much activity was seen in each one.

Thank you very much Olga and Reid, this is a huge effort and we really appreciate your work so far.  I eagerly await seeing the final 25-30 page summary.

Update on the Metrics

We have been working with the Mozilla Metrics team to surface information that can help us better visualize macro trends as they play out in Bugzilla.  This will help us measure change over time and see problems coming before they reach a critical point.  To help deal with the massive scope of the information stored in Mozilla’s Bugzilla, we are copying some of this data into an Elastic Search database that only includes the meta data of each bug.  This lets us do massive queries without hindering production by bringing Bugzilla’s DB to a standstill.  Progress on this work has been slower than I would like but we are making gradual progress.

We are working to make this data public via a REST interface but in the mean time, for those that have LDAP level access at Mozilla, you can see some of the visualization prototypes we have been working on at the following link:

https://metrics.mozilla.com/bugzilla-analysis-site/

Several of these prototypes are in use in production already.  However, work is still ongoing and here is what stands between us and a public REST interface:

1. Complete schema modification to make sure that the database can handle queries relating to review queues.

2. Insure that proper automated testing is in place to verify that the data in the ES database is consistent with that found in Bugzilla’s DB.  We have seen issues here so this is important.

3. Address the remaining security concerns around the meta data associated with open security bugs.

None of these are that large in scope, however, the metrics team has been very tied up and has had difficulty finding time to work on them.  Last week saw a jump in progress on the first item and I’m hopeful we will see that crossed off the todo list soon.  I have also attended very constructive meetings that show this rising in priority for the Metrics team which is also a positive sign we will see this done sooner than later.

Project Wiki Page: https://wiki.mozilla.org/Bugzilla_Anthropology

What is this all about?

The goal of the program is to take a deep look at how people are currently engaging with Bugzilla. We want to see what we can learn about the tool’s good, bad, and frustrating qualities. You can’t look at Bugzilla without understanding how people work. The research is also surfacing information on the life cycle of bugs and how people interact with them along the way. The primary goals are as follows:

• Identify the information we want to gather.
• Interview stakeholders across the organization and community.
• Surface statistics that are relevant to its use.
• Identify best practices, useful hacks, tools, and key problem areas.
• Provide an overview of how Bugzilla is being used.
• Complete the research by the end of January 2012.

How are we going about doing this?

In an effort to get to the root causes of issues and find the hidden best practices, we are gathering data in two ways. First, we are conducting 1 hour interviews that probe into various topics focusing on how people interact with bugs throughout their life cycle.

The second method is statistical. We are diving into the considerable dataset we have stored in Bugzilla’s historical log. We hope to surface trends that help point out problem areas where we can improve as well as shed light on what’s working well so we can highlight those successes.

What have we learned so far?

We have just completed 10 interviews and posted the raw transcripts last week. We held off posting this material until we felt we had the absolute minimum representative coverage. The transcripts are raw feedback, they have been verified by the contributors, and they have all volunteered, in true Mozillian fashion, to put their names on them. Thank you to all the brave contributors that went first in the hopes of improving things for all.

Interview Transcripts: https://wiki.mozilla.org/Bugzilla_Anthropology#Interviews

We have also been working with Metrics to gather data on Bugzilla usage. Much of this data is still behind layers of security as the tools that help visualize trends are still prototypes. Once the tools are ready, the security group will need to review the system as it includes statistical information on security bugs. We will need to work with them to double check that we aren’t leaking out sensitive material before reducing the security to LDAP or preferably making it publicly available.

How can people contribute?

The best way to help is to review the data, give us feedback on a new set of interview candidates, and anything new you think we should be exploring. We intend to do another 10 interviews as the next phase. We will be including more community members in this round as they are currently under represented. The current list of questions we are asking and who is on our radar as candidates can be found at the following Etherpad. Feel free to add suggestions here:

Interview Etherpad: https://etherpad.mozilla.org/BugzillaInterviews

Once we start to surface the statistical side, I will write up another post.

If you have any questions you would like to see added to the interviews, have any feedback on the project or results that you would like to share, you can find me on IRC in the #pm channel, email me at mbest@mozilla.com or comment below.

Cheers,
Martin