Jay's Blog Thoughts on Mozilla marketing, err…engagement.

One of the main reasons for conducting our QA survey was to gather feedback from contributors to find out what they think works (in terms of events and activities) and what they want to see the Mozilla QA team do to grow, empower, and improve the community going forward.  Following Part 2, which explored participation, we now look at some direct input from the community to wrap up the series in Part 3.

The following 3 questions provided plenty of data and ideas for the QA team to consider as we move forward…

What are/were your biggest obstacles to working on the Mozilla project?

Anyone can download Firefox and use it or “test” it, but what comes next if you want to get involved with the Mozilla project?  For potential contributors, that’s not always an easy question to answer.  The data below represents responses from 93 volunteers…they were asked to select their top 2 obstacles.

Though the Mozilla project is very open and there are plenty of opportunities, it can be a daunting task to figure out where you fit in or how to get started… and the majority of those that responded agree:

With so many wiki pages, newsgroup posts, blogs, and forums to comb through to figure out what is going on, the QA community can easily get lost.  That is why I have been busy working on the QMO (quality.mozilla.org) redesign project.  We hope the new QMO (coming Summer 2008) will provide a launch pad for anyone that wants to get started with Mozilla QA.

The Mozilla QA team can also address two other big obstacles with a combination of a well organized QMO site and more collaboration with the community.  While the site itself will provide better documentation and tips on getting involved, the team  hopes to be more proactive in our communication with contributors.  Our goal is to build relationships that will provide the mentorship and guidance that many people need when they first join the community.  QA does not have the benefit of a module ownership system (like developers) and therefore it is up to the QA team to be more visible and step up to become leaders within the community.

There are also a few obstacles, like time and software/hardware needs, that we can’t always remedy… but we are trying to help as much as we can.  Many contributors have already benefitted from Mozilla’s Community Giving Program and the QA team is always looking for better ways to schedule our events and projects around our international community so that people in all time zones have a chance to participate.

With a better understanding of some of these stumbling blocks, we can better prioritize our efforts and continue to work hard to make it easier for people to get involved and help them become effective contributors to the Mozilla project.

How do you think the current Bug Days and Test Days can be improved?

Bug Days have been around since the early days of the Mozilla project when Asa helped organize weekly bug triage events, a tradition we continue every Tuesday.   Test Days are every Friday and encourage more community test coverage for critical bug fixes, imporant milestones and new releases.  We have had decent success with both events over the past few years, but know we can do better with input from the community.

There are some common themes in the feedback we received.  I have summarized a few of them below:

1. Different days, better time slots
2. More interaction with QA team
3. Focus on helping newbies
4. Plan specific Bugzilla activities, like the unconfirmed bug triage projects.
5. Better documentation and tutorials on some basic QA “lessons”
6. Improved feedback/reporting channels
7. Allow localization of content and tools
8. Make it easier and faster to participate and accomplish tasks/goals
9. Weekend long events, open scheduling, etc.
10. Innovate communication and collaboration with community

The Mozilla QA team is actively working on a few of them now, and we have definitely either thought about or have plans to address the rest.  Now that we have a basic idea of what the community wants to see, I plan to contact a few of the contributors to get more details and then work with the QA team to make them happen!  This is a great start to improving our collaboration with the community, and I can’t wait to see some of these improvements in 2008.

Imagine you are visiting QMO (quality.mozilla.org) for the first time. What would you expect to see?

The “alpha” experiment with the current QMO site has provided good lessons and inspired great ideas for the “official” QMO site, but I also wanted to get some realistic expectations directly from contributors to make sure the design direction I have taken with the visual refresh and site architecture will meet the needs of our diverse community.  Below are a few thoughts directly from those that participated in our survey:

•  Tests that can be run without using Litmus, a functional extension to interface with QMO
  • DONE!  See the Mozilla QA Companion
• Instructions on how to collaborate and perhaps FAQ section.
• Events, Documents about how QA works/getting involved, contact methods.
• Results from the work done. (counters of bugs squashed during a bug day etc) (Metrics or statistics)
• Link to the latest beta version and to the latest nightly version of each
  branch. Explanation of branches and version numbers for branches.
• A short description about the website, its objectives and role in the Mozilla
  community
• Have a download link for FF3.0 Beta. Do you actually read all these comments?
  • YES, and thank you for the idea, see the very top of QMO. We have been doing that since Beta 2.
• Very real as I haven’t been there yet. I would expect to find a nice UI (last priority), but lots of documentation on how to be a good QA and QA Community Member (E.G. what to do and how to respond to questions and what to look for) and updated lists for what is the top priority like do you want to get rid of a lot of bugs this month of confirming new bugs (E.g. the QA staff can confirm). Or do you want us to push Firefox and file as many non-dupe bugs, that kind of list. I would also want to find a portal or page that lists all of the Firefox IRC channels, how to get in touch with the devs and QA staff and especially a list of the sites like I didn’t know there was a test day site/blog would be nice
  for all of these sites to be well documented in one place for easy resource access and such.

I wanted to throw that last one in there because it’s comforting to know that everything I have gathered so far about the needs of the community is pretty close to what people that have never visited the site might expect to see.  Some of those other ideas will also help me refine the layout and prioritize the content we need to create for the site.  It is awesome to know that this feedback will have a direct impact on the success of the new QMO when we launch this Summer.

That’s all I have in terms of the feedback we have collected.  The QA team has a lot to think about and hopefully when the Firefox 3 release madness is over we will all be able to dedicate some time to addressing the issues and exploring the new ideas we have collected through this survey.

To wrap up this series, I wanted to share with everyone a few responses to the question below to remind us all why we do what we do as part of the Mozilla community…

Why do you dedicate your time to help with the Mozilla project?  What about Mozilla QA keeps you engaged in the community?

• I spend a lot of time using Firefox, as a user or Web developer. That’s why I’m interested in contributing back and helping make it even better.
• I want to support a product I use daily and love.
• I get some sort of sick pleasure out of it.
• I love mozilla* but I hate bugs in it
• It’s FLOSS, so I can.  The browser is a very important piece of software, probably the most widely used application.  The net/web is where it’s happening.
• I like to improve Firefox and as a future programmer, also learn about the software development process.
• I like firefox. It saves me from having to use IE 4/5/6/ and 7.
• Another browser needs to exist to put pressure on Microsoft, it drives innovation.
• To help Mozilla produce great quality products and the QA team did great job in hosting QA events.
• I was and I’m involved in many testing programs from software to hardware. I’d like to know new technologies and applications and I use this to improve my professional training and to be prepared for the next
• I’m a Firefox fan since Phoenix 0.3, got addicted some way.
• Because there are many people out there that need to be protected and updated and i love to help others achieve this goal
• So many people devoting time, working together (sometimes/always?) without pay. it just feels like its a project by the community for the community.
• Lots of people use Mozilla products, especially Firefox, and it’s very rewarding to be part of something so big and significant for a lot of people.
• Connected to my class at school, but also it’s a great community and a positive learning experience. since becoming more involved with mozilla i have learned a lot about real world development.
• Ive got nothing better to do

There’s a lot more where that came from, along with all the details from the survey results… so if you’re interested, check out the complete report online or download the pdf.  Some of the charts and tables aren’t very useful (limitation in the surveygizmo reporting), but all the data is there.

I hope this series of blog posts provided some interesting information about the Mozilla QA community and improved our visibility within the Mozilla world.  People sometimes forget that there are many QA contributors out there helping out that you won’t see in any Bugzilla comments or CVS checkin logs.  They are sometimes invisible to the rest of the community, but their contributions are just as significant to the Mozilla project and critical to the success of products like Thunderbird and Firefox.

My goal is to make sure the Mozilla QA team and community never feels invisible…

In Part 1, we learned a little about our contributors and their test environments.  As I continue my series of posts covering the results of the QA community survey, I wanted to shift the focus to volunteer participation in Part 2.

The following questions revealed a bit more about what people are most interested in doing and how they currently contribute to the Mozilla project.  By understanding how current contributors are helping out with the Mozilla project, we can get a sense of what types of people fall into certain roles or responsibilities.  That is extremely valuable as we continue to grow as a community and bring in new people that may not always know where to go or how they fit in.  It’s up to us to guide them.

Let’s take a look at some of the questions and data…

Commitment

How many hours per week do you spend on Mozilla QA related activities?

I currently spend…

I wish I could spend…

Those two bar graphs tell me a lot about the dedication and commitment our community has for the Mozilla project.   It’s quite amazing that most of our QA volunteers spend at least 1-3 hours a week helping out.  But what’s even more amazing is that a lot of them would be willing to spend anywhere from 4-10+ hours if they had that time!  That is why the Mozilla project has been so successful and why the quality of Firefox continues to improve daily.

Participation

What Mozilla QA related activities are you working on right now?

There were many open text responses for this one, so I’ll just share a few that reflect the variety of activities people are involved in:

• Beta Testing and Nightly Testing
• I have a more developer oriented profile. I do some bug triage, and help
  fixing by writing patches sometimes.
• Memory Performance, Bugs
• Confirming Bugs, Litmus test cases
• Try to reproduce what people are telling at bugzilla (Firefox only) and to
  gather additional info about the bug, like regression date etc. I’m not a coder
  and I only do things that are fun to do.
• Working on the FTR – For The Record; Mozilla’s media watch.
• Testday/Bugday support, QA Community Involvement, Seneca Club Moz,
  Bug Triage
• Looking into Norwegian translation

As you can see, our community is covering a broad set of QA activities… including general testing, bug triage, localization review, community leadership, and developer support.  As we learn more about specific areas that some of our volunteers are interested in, we’ll be able to better spend our time organizing events and projects for them.

What Mozilla QA related activities are you most interested in?

The survey listed a few activities and gave volunteers three options for their level of interest:

1. Definitely
2. Maybe (with help)
3. Not for me

The pie charts below show the distribution of interest for 4 major QA activities:

No surprise that “Testing” was the most popular, as it is the easiest way to contribute.  And really, if you’re using Firefox daily and take the time to provide feedback or log bugs, you’re already testing it.

Quite a few people also showed interest in working with “Bugs”, but it’s clear that we need to do a better job educating them about Bugzilla and helping them with bug related tasks.  Bugzilla is the center of the Mozilla universe, so the Mozilla QA team and community definitely should use it more often and take advantage of it’s many features to improve QA processes and product quality.

“Feedback Analysis” represents general Hendrix feedback and Talkback/Breakpad crash data analysis.  The survey included both separately, but the results were very similar, so I put them into one bucket.  We still have a majority of the volunteers interested in helping with this, but again the QA team needs to work with them to figure out interesting ways to look at the data and guide them as they comb through tons of user feedback and try to make sense of it all.

Then we have “Automation”, which wasn’t as popular as some of the other activities.  Not too surprising, considering that working on test automation requires specialized skills and programming knowledge… and from the survey data, it looks like most of the volunteers that responded were just getting started with the Mozilla project and were mostly into general testing.  That’s ok… we still have 33% that just need some advice and mentoring to get their hands dirty!Some write-in activities were “Documentation”, “Debugging”, and “Localization”… all of which are all very valuable contributions.  Mozilla QA will definitely need help with all three when we start developing the new QMO site this Summer.

Preferences

Which Mozilla QA mailing lists are you subscribed to?

One of the most successful Mozilla QA mailing lists we have had in a long time is the betatesters@m.o list.   Marcia and I created it last year to build a community of dedicated testers to help with alphas, betas, and pre-release builds.  The list now has over 1600 members, so it’s no surprise that over half of our volunteers that responded are subscribed to it.  Coming in second is our tried and true qa-community@m.o list, which we have had for many years.  I think both of those lists are going to be the QA team’s priority for keeping track of progress within the community and as major broadcast channels for big announcements and calls for help.  But there are so any ways to stay connected, so it really doesn’t matter which list you’re on… we’ll reach you somehow.

As a member of the Mozilla community, what are the roles that best categorize your involvement with Mozilla? (Please select up to 3)

This question was intended to provide us with some idea of how contributors saw themselves within the community.  One of the biggest challenges for the Mozilla QA team is to find the “right place” for some of the volunteers that come our way,  so data like this definitely give us a good sense of where the popular spots are and hopefully with the feedback we have gathered, we’ll be better prepared to help new contributors find their way.

Some volunteers fall into multiple categories, as seen in the table below, which contains data from 93 responses.  It’s interesting to see which buckets people put themselves in.  Obviously most people see themselves as Firefox users… but it is great to have a healthy 34% of folks testing nightly builds as well.  Another point worth noting is that 20% of the responses came from Web developers, one of the most valuable segments of the community when it comes to QA, as they are usually the people that are up to date on the latest fixes and their impact on various technologies and web sites.

Overall, it looks as if we have nice diverse group of folks making up the QA community.  That definitely helps us get the most coverage, especially when we have nightly builds, Firefox 3 betas and security releases for 2.0.0.x all going on at the same time!

Part 3 coming soon…

I hope my first 2 posts have provided a good snapshot of what types of people make up the QA community and the various activities they participate in.  The data we have gathered is meant to give us a base to work with as we try to find better ways to engage new contributors and improve our processes and tools to encourage more collaboration.  Next time we will take what we have learned so far in Parts 1 and 2 and try to make sense of the direct feedback we have received through the survey.  There are a lot of great ideas and requests, so it will be nice to share that with everyone.  Part 3 will focus on bringing all of that together to create action items for the QA team to improve community efforts in 2008.

As part of our QA Community strategy for 2008 I designed a volunteer participation survey in late 2007 as a first step in better understanding our QA contributors.  The survey was available online for about two months from Jan 2  – Feb 27, 2008 and we reached out to contributors via newsgroups, mailing lists, and QMO.  In the end, we had a pretty good sample of 114 responses.

The goal of the survey was to provide the Mozilla QA team a snapshot of what our community looked like and to  gather feedback from volunteers so that we could better plan and execute on our community events and projects going forward.

There is a lot of data, so I plan to break up the results into 3 parts.  Part 1 covers information gathered about the contributors that make up our Mozilla QA community.   I will follow up with Part 2 soon, which will cover how the contributors currently participate in various activities and their preferences on ways to stay involved.   Then I will wrap it up with Part 3 to go over the feedback we collected and discuss what the Mozilla QA team plans to do in 2008 with the knowledge gained from the survey.

Part 1: Community Profile

Usage vs Participation

Though many people have been using Firefox for years, it’s interesting that most of the volunteers that responded have only been active Mozilla contributors for less than 2 years.  It’s good to know that our efforts continue to convert fans into curious community members.   The challenge is to help them find their place within the community and enable them to become regular, long-term contributors.

How long have you been using Firefox? (including Phoenix/Firebird if you know what those are)

Years       Count    Percent
3 – 4 years    49    46%
5+ years       37     35%
1 – 2 years        18       17%
< 1 year              3         3%
Total Responses:    106

How long have you been an active participant in the Mozilla community?

Years       Count   Percent
< 1 year        46    49%
1 – 2 years    16     17%
3 – 4 years       16      17%
5 – 6 years         9      10%
7+ years            7        7%
Total Responses:    93

Continue reading →