Category Archives: Forums

The all new SUMO

Today we are going to make one of the biggest changes yet to SUMO, the Mozilla Support site, and this blog post is about what changes we are making. The changes will effect you the most as a user, but there are a number of changes for contributors as well.

First, a little history, what’s the SUMO team been up to this year?

For the last 9 month the SUMO team has been working on a new way to let users access our site content. Until recently the only way to reach most of our articles was by search, or by following links in articles. This is how wikis traditionally work. Of course that way of accessing content only works for a part of our visitors, some people want to search and some people want to browse to the solution, drilling down with ever finer topics to reduce the number of article that are related to the issue.

To come up with a new information architecture that would let people drill down like that, we first researched the mental model of our site users, how they think about issues and in what categories they would look for them. Based on that we created a small number of base categories and assigned our articles to those categories.

The next step was figuring out how to make this information architecture visible. We started to lay out a number of alternatives on paper and tested with real people in a lab. This paper prototyping gave us a way to test a number of ways to lay out the information very quickly. After a number of iterations we settled on the final designs and workflows.

Now we had everything to start adapting our site, but since this would be a big redesign, and we’d soon switch to the new unified One Mozilla design anyway, the decision was made to use this opportunity to rebuild the site based on the new theme, and that’s why the changes today not only affect the KB, but every part of our site.

So, what is changing? What does it look like?

The main change is, that we now support several products from one start page and all articles can be accessed by browsing. Let’s start with the start page:

We have the main topics on top, they allow you to start browsing by selecting your issue first, and then the product you have issue with.

One step below you can see the hot topics. Those are actually articles, things that came up recently and affect a large number of people. By providing them upfront we save a large number of people the hassle of searching or browsing for their solution.

Below that we have the product picker, this is a way to navigate our content by choosing the product first and then narrowing down the topics.

No matter what way you select, topic first or product first, you’ll end up narrowing down the number of articles to a scanable few and proceed to read one of the articles.

The important thing for localizers to note is: all of this is automated, there is no need anymore to create navigation pages and all the confusion that brought with it.

So, how did the article view change? On the surface not much has changed, but because we keep track of topics, we can now offer you a way to move to related topics, which is particularly interesting for people landing on articles from external searches:

Much, much more has changed, but this is the gist for the KB part of the site.

So, what has changed for forum contributors?

While the new iA did not touch the support forum per se, we took the redesign as an opportunity to improve a number of factors in the listing of questions for our contributors.

The new design is more friendly and clean, but at the same time gives more information about the thread content at the same time. This is especially helpful when contributors scan the thread listing page deciding which thread to pick next.

We already started rolling the design out to our contributor base over the last week and will start rolling it out to 1% of the general audience today. If everything goes to plan we’ll make it available to the general audience on Monday. If you want to try it out now, just register an account, and if you have any feedback, please use the comment section below.

The new information architecture will open up our content to a whole new group of users and make it much more accessible, while our new design is more coherent, taking into account all of the features we added since our first release while also being consistent with the Mozilla sites in general. All of this makes us very excited and hopeful that we’ll get that much closer to our number one goal: Happy users!

The new magic SUMO tool for frequent questions, and an interview with its developer, Tobias Markus

For a few weeks now we have been replying to almost every single question that is asked in the Mozilla support forums. That is up from about 50% last year and it shows the dedication of our community. Also over the last few month we have made a number of changes to SUMO to reduce the flow of simple questions that have answers in the KB already. That has been quite succesful, not only are we getting really hard questions now, but they are solved at a higher rate than before too.

The challenge now is that we have to ask for more information before we can solve an issue. Repeating those questions is quite tiring, so we decided to build a tool to help out our community and gather all the important questions to ask in one place.

We had actually planned to work on it later this year, but one of our great community members, Tobbi, saw an opportunity and offered to try his hand on it.

The result went online a few days ago, and is amazing. Sign into SUMO, and check it out for yourself:

Chose a question
Click on the response tool (the magic hat symbol)
Select an answer or a request for specific information
Customize as necessary and send!

Over time this tool will hopefully gather all the shared wisdom of our community and help us solve issues in as few steps as possible.

Interview with the developer, Tobias Markus.

This is not the first time a community member has been helpful in the development of Kitsune, the software that powers SUMO, but it’s the first major feature. We’d love to see much more of this, and I asked Tobbi for an interview to ask him how he got involved for anyone who wants to follow in his footsteps.

Tobbi, you have been a SUMO community member for quite a long time. Where did you start and how was your SUMO journey so far?

It was late 2008 when I started contributing to SUMO. My portal to the SUMO world was the Firefox LiveChat, which I joined as I really liked the personal touch that the chat had. Later on, I found out that there was a German localization group for SUMO articles, so I started translating articles into German and helped reaching the goal of then 100% localization coverage. I even gave forum support, although I preferred LiveChat because it was more personal than the forums.

How did you get the idea to help with the development of SUMO?

The idea for development started when I was working on a Firefox add-on together with Propeng. We implemented various features for the old forums to make them more contributor-friendly and add some tools that aimed at making life for contributors easier. However, due to the amount of unfinished code and the switch to Kitsune, the project was stalled. That’s when we decided to split the big project into smaller chunks to make them compatible again with the new forums. One of these smaller projects was the Knowledge Base article autocomplete that was landed a few months ago. I think it was Rosana who suggested integrating the Knowledge Base autocomplete into the Firefox support website directly. That’s when I started contributing to the development of SUMO.

Did you get any help with your project?

The SUMO team and other contributors really helped me a lot. At first I was posting an early draft to the contributors forum to gather suggestions from other contributors and the team. The feedback I got was huge. I spent the next couple of weeks reacting on it and making a lot of improvements to the code to enable things like instant preview, edit mode, and even flags that can limit certain responses to certain groups of contributors. The SUMOdev team also answered some implementation questions.

Once this was done, the next thing was to make the experience perfect for contributors. I brainstormed a lot with Bram [the SUMO UX designer] about best practices in order to get a good user experience design and spent some more weeks making the necessary changes.

The fourth step was a final code review by Ricky Rosario. That went smoothly as well, the necessary changes I had to do were minimal and most of them were about simple code style fixes. Once these were fixed, the code was ready to land.

The final step was to create the canned responses. I asked the community again to get back to me with some frequently used responses and the amount of responses I got was overwhelming. I chose around 10 of the most frequently used ones (based on my own judgement) and rewrote them a little. Kadir then helped me put each response in its own article and create a main article for the category structure that has links to all the individual responses.

What is your advise for people who want to get involved with SUMO development?

Drop by in #sumodev on irc.mozilla.org and say hello. The SUMOdev people answered any questions I had so that canned responses could become a success. Also, there is good documentation available online about setting up a local copy of SUMO. So, if you have a great feature in mind for the support website and want to start developing, give it a go!

Tobi, thank you for this interview, and in the name of the SUMO community: thanks for making everyone’s live so much easier!

Firefox support questions: on our way to 100%

Late last year, we set a goal for 2012 to achieve a 100% response rate in our community-powered support forum. In other words, every user who asked a question in our forum should get a response. It felt like an ambitious goal when we set it, because only six months ago, we averaged around just a 50% response rate.

Average response rate has increased from ~50% to ~90% in just a few months.

A lot of things have changed since we set the goal, and they all contribute to this dramatic improvement in response rate:

Our community has grown bigger and is a lot more active. This makes answering questions more fun than before.
We have started to host bi-weekly SUMO days focusing specifically on answering forum questions. Initially pioneered by Michelle, nowadays they are run by Madalina.
We have improved the start page by introducing basic browseability that we’ve previously lacked. This has reduced the number of new questions in the forum and is part of the massive project to revamp the information architecture of SUMO to significantly increase the number of users who can help themselves on our site.

The increased response rate has a number of other side-effects as well, such as an increased chance of the user getting back to us to let us know if the answer solved their problem. The solved-rate has more than doubled since November last year.

This is just the beginning. We have yet to fully roll out the revamped site and information architecture mentioned above. We also just introduced some new cool features to the forum that makes it easier to see how many questions are still unanswered:

SUMO regular Tobbi is working on a Canned Response feature that will make it much easier to quickly respond to questions. You can watch a screencast of the work in progress. It’s already totally awesome!

Big thanks to everyone in the SUMO community who work hard to respond to our users’ questions! Without you, this wouldn’t have been possible.

Speaking of the forum… what are you waiting for? It’s SUMO Day today, so head on over and answer some questions!

The 8th SUMO sprint

Last Tuesday we finished the 8th SUMO sprint of this year. I’s been a short sprint, because for half of it Mozilla web developers met for a work week in California. Nevertheless we were able to land some great improvements to SUMO.

One of our top priorities right now is improving our search results. To that end we started an initiative to rate KB and forum results as equals, based solely on the content instead of showing KB articles on top and forum posts at the bottom. We hope that this step will result in better search results by higher click through rates. This is a project that will take more than one sprint to be implemented. We completed the first of 3 phases in this sprint and are planning to implement the rest over the next sprints. You can read Will’s bug comment about it, if you want to know more about the gory details of the unification work.
We were also able to implement a feature in the forums that gives our community an indicator for how we are doing and shows everyones impact on the forum: We call it the forum feedback indicator. It shows you how many questions were asked in the last 3 days and how many have a reply. It also links to the questions that haven’t been answered yet. The progress bar is orange, for any values below 100% and turns green when all questions are taken care of.

As with every sprint we also fixed a number of annoying bugs, and you can always follow along on the excellent scrumbugs. Here is the list for the 8th sprint.

However I’d like to point out one specific issue, that was bugging us in the forums for quite a while, creeping back when we thought we had squashed it: In some cases, a question displayed “No replies”, even though it already had replies. That bug is now hopefully fixed for good.

SUMO Thread analysis: Better Tools

At SUMO we always want to help our contributors help our users. To further this goal, we have begun analyzing SUMO threads, to see how we can help contributors improve their responses, and hopefully help more users. To help this goal, I and some other members of the SUMO team have analyzed a week’s worth of threads to try to find holes in our current process.

The Process

We analyzed all threads created on SUMO from April 1 to April 7. There were 365 total threads for this time period. We then arranged these threads into categories, 41 to be precise. These categories range from “Website Looks / Acts Wrong” to “Firefox Crashes” and everything in between.

Now, out of these categories, 23 of them (or 56%) have less than 5 threads per category, which was too small for analysis. Another was because of the Java blocklisting, which while it may be interesting for another analysis, for this case we just discarded those threads.

Then, we chose those categories that had the most useful information for our purposes. How many threads had responses, how many were useful responses, how many were just general replies (like “Try in Safe Mode”), etc. These threads turned out to be the following:
Firefox Crashes (19 threads)
Website Looks /Acts wrong (41 Threads)
Firefox is Slow (10 Threads)
Problems Caused by Plugins (10 Threads)

The Results

Out of all these threads, it became apparent that over half (50-60%) of threads have just very basic, general troubleshooting answers. These threads were also the ones that had very low (~20%) reply rate from the original poster. It seems from this that users want to reply when a contributor addresses their issue more directly than when there is just a general answer.

The answers that we found were not General Answers or Solid Answers were as follows:
Needs better Troubleshooting (roughly 12% of questions)
Too technical (Roughly 7%)

The rest of the questions that did not fall into these 3 categories were those with actual solid answers. While they may not have had a solution marked, they did have answers that, from how well they were written and how they addressed issues relevant to the original poster, seemed that they would have a high chance of fixing the problem. If we can increase the rate of users coming back to SUMO to update their questions, that will help the number of “Solid Answer” threads go up.

One good thing we found in our analysis was out of the 4 main groups of questions, only one question did not have an answer! This means we are doing really well on making sure we reply to 100% of threads.

Solutions

From the analysis, it seems that if we can help contributors provide more useful answers, we should begin to notice a higher percentage of solved and solid answers. To help with this goal, we have come up with a few different suggestions that we can begin to implement immediately:

Contributor guidelines:
Provide documentation on SUMO for contributors. This can range from how to begin diagnosing different issues (crash IDs, extensions, websites, etc.) to just general help for interacting with different users. We can give common issues, how to reply to them, tools to suggest to users, tools not to suggest, etc.

Contributor Workshops:
Beginning Class: Once every X weeks we have a class to teach people who want to contribute or have recently started contributing the basic ways to respond to threads, troubleshoot, act professionally, etc. These don’t even have to be ran by Mozilla Staff, experience and trusted contributors could be asked to help run these.

Special Guest Class: Developers, SUMO Staff, etc. can come and have a webinar to explain new features in firefox, how they work, what some common issues may be or are, and types of feedback they are looking for. Example, a Firefox dev in charge of the pdf.js feature could have a session about what it is, a basic overview of how it works, some known issues, how to fix them, and asking the community to keep an eye out and give feedback on X Y and Z. Then have a Q & A Time.

Specialty Webinars: Every so many months, or as needed based on feedback, the SUMO Staff gives sessions on diagnosing Hangs, how to read a Crash ID, website troubleshooting, etc.

Help Wanted!

Now, all of these are just ideas, for now. Obviously the sooner we get better tools to the community, the sooner we can improve the service we give to End-Users. We would love to get feedback from the community on ways they think that we can improve the currently available tools. Nobody knows ways we can help the community better than the community, so the more input we can get from you, the better! You can ping me on IRC (:Tyler), send me a message on SUMO (tylerdowner) or leave a comment. I’d love to hear from you and hear all your ideas! Specifically we would love feedback on these areas:
1. How useful is this current analysis, do you want to see more information from it, should we repeat it, if so how often, etc.
2. Do you feel these tools will help the community (you!), do you have suggestions, or even totally new ideas?

SUMO sprint 2012.4

This is a quick update on what we are currently working on in the 4th SUMO sprint of this year, the results of this sprint will go live between February 28th and March 6th.

Our dev goals for this quarter were to get the SUMO KPI dashboard up and running, and to switch to a new search engine that would allow us to implement all the improvements suggested by the ongoing taxonomy/ux research and the usability research before that.

We have recently turned our new search engine (Elastic Search) on for 50% of our site’s users, and the early results are very promising. The Click Through Rate, our main success metric at this point, of the new engine is already better than that of the old one. Currently we are gathering more data to get reliable results, but if the final results agree with our preliminary findings, we’ll switch to Elastic Search 100% in early March and start improving our results. One big issue we’ll tackle soon is the performance of Elastic Search, currently it is much slower than our old search engine. However Will already has ideas that will hopefully give us significant improvements in this area, starting with sprint 5.

Our KPI dashboard is nearing completion, thanks to Ricky. We already have many of the metrics we set out to present and few are left. This sprint will give us the Army of Awesome numbers, the number of unique visitors we have at SUMO and our localization coverage.

One change that seems small, but will hopefully have a big effect, is the change of our font, colors and sizes. We had numerous complaints that our text was hard to read and links were easily overlooked. Bram, our UX designer, and font expert extraordinaire, came up with a new style that uses stronger colors for links, changes the gray on white of our body text to a darker black, and moves us from serif to sans-serif. The latter change is significant, there is a large body of research on what is better for readability serif, or sans-serif, and amid a lot of myths there are arguments for both sides. However it seems clear that there is only one factor that reliably affects readability: familiarity. Of the top 100 websites today only 4 use a serif font for body text. Since users today are so much more familiar with sans-serif our hope is that this change will contribute to the overall readability of SUMO.

Another thing we have been working on is several customizations of our contributor forums, so they could be used by the Firefox Aurora and Beta communities. Expect more activity in those forums very soon.

As always, there is a good number of new features and fixes for smaller bugs:

ID	Summary
730065	Allow sort by original post date
718813	let people add screenshots when asking a question
726037	Combine some of the KPI dashboard graphs
645546	How to contribute articles overview in the dashboard
724833	Change AoA signposts
625841	Reword email notifications for Questions
720226	We need reviewing rights for en-US and other locales
724483	remove en-US part from link in ‘ready for localization’ emails
725287	Use One Mozilla breadcrumb style (revert bug 716018)
726112	Allow entering a message for recent contributors, even if you approve your own revision.
722697	Show contributor sidebar in the questions app
726856
706948	In some cases, a question displays “No replies” with a reply below
722509	notification about private message has localized URL
728307	Please add {for} support for fx12, m12 and add Firefox 12 to the dropdown UI, remove Firefox 9 from the UI

Matt Brubeck on answering support questions

Download video: MP4 format | WebM format

Bottom line – answering support questions is good for you! Get a SUMO account and then get started. Also, check out this video which shows you how to filter support questions by tag.

SUMO Blog

The Mozilla support project

Category Archives: Forums

Matt Brubeck on answering support questions