SUMO Thread analysis: Better Tools

At SUMO we always want to help our contributors help our users. To further this goal, we have begun analyzing SUMO threads, to see how we can help contributors improve their responses, and hopefully help more users. To help this goal, I and some other members of the SUMO team have analyzed a week’s worth of threads to try to find holes in our current process.

The Process

We analyzed all threads created on SUMO from April 1 to April 7. There were 365 total threads for this time period. We then arranged these threads into categories, 41 to be precise. These categories range from “Website Looks / Acts Wrong” to “Firefox Crashes” and everything in between.

Now, out of these categories, 23 of them (or 56%) have less than 5 threads per category, which was too small for analysis. Another was because of the Java blocklisting, which while it may be interesting for another analysis, for this case we just discarded those threads.

Then, we chose those categories that had the most useful information for our purposes. How many threads had responses, how many were useful responses, how many were just general replies (like “Try in Safe Mode”), etc. These threads turned out to be the following:
Firefox Crashes (19 threads)
Website Looks /Acts wrong (41 Threads)
Firefox is Slow (10 Threads)
Problems Caused by Plugins (10 Threads)

The Results

Out of all these threads, it became apparent that over half (50-60%) of threads have just very basic, general troubleshooting answers. These threads were also the ones that had very low (~20%) reply rate from the original poster. It seems from this that users want to reply when a contributor addresses their issue more directly than when there is just a general answer.

The answers that we found were not General Answers or Solid Answers were as follows:
Needs better Troubleshooting (roughly 12% of questions)
Too technical (Roughly 7%)

The rest of the questions that did not fall into these 3 categories were those with actual solid answers. While they may not have had a solution marked, they did have answers that, from how well they were written and how they addressed issues relevant to the original poster, seemed that they would have a high chance of fixing the problem. If we can increase the rate of users coming back to SUMO to update their questions, that will help the number of “Solid Answer” threads go up.

One good thing we found in our analysis was out of the 4 main groups of questions, only one question did not have an answer! This means we are doing really well on making sure we reply to 100% of threads.

Solutions

From the analysis, it seems that if we can help contributors provide more useful answers, we should begin to notice a higher percentage of solved and solid answers. To help with this goal, we have come up with a few different suggestions that we can begin to implement immediately:

Contributor guidelines:
Provide documentation on SUMO for contributors. This can range from how to begin diagnosing different issues (crash IDs, extensions, websites, etc.) to just general help for interacting with different users. We can give common issues, how to reply to them, tools to suggest to users, tools not to suggest, etc.

Contributor Workshops:
Beginning Class: Once every X weeks we have a class to teach people who want to contribute or have recently started contributing the basic ways to respond to threads, troubleshoot, act professionally, etc. These don’t even have to be ran by Mozilla Staff, experience and trusted contributors could be asked to help run these.

Special Guest Class: Developers, SUMO Staff, etc. can come and have a webinar to explain new features in firefox, how they work, what some common issues may be or are, and types of feedback they are looking for. Example, a Firefox dev in charge of the pdf.js feature could have a session about what it is, a basic overview of how it works, some known issues, how to fix them, and asking the community to keep an eye out and give feedback on X Y and Z. Then have a Q & A Time.

Specialty Webinars: Every so many months, or as needed based on feedback, the SUMO Staff gives sessions on diagnosing Hangs, how to read a Crash ID, website troubleshooting, etc.

Help Wanted!

Now, all of these are just ideas, for now. Obviously the sooner we get better tools to the community, the sooner we can improve the service we give to End-Users. We would love to get feedback from the community on ways they think that we can improve the currently available tools. Nobody knows ways we can help the community better than the community, so the more input we can get from you, the better! You can ping me on IRC (:Tyler), send me a message on SUMO (tylerdowner) or leave a comment. I’d love to hear from you and hear all your ideas! Specifically we would love feedback on these areas:
1. How useful is this current analysis, do you want to see more information from it, should we repeat it, if so how often, etc.
2. Do you feel these tools will help the community (you!), do you have suggestions, or even totally new ideas?