Category Archives: about:sumo

SUMO now helping an additional 7.3 million visitors

About 3 weeks ago we made the switch to a new information architecture and new design. The goal was to improve the browsability of the site and help people find the articles that they were looking for. 3 Weeks later we can now take a look at our key performance indicators to determine whether the whole project was worth the effort

Methodology

Since this project was primarily concerned with the Knowledge Base, we can focus on the helpfulness rating in this channel. Also, we know from our exit surveys that about 80% of our visitors use the KB.  The KB helpfulness rating is based on the survey that accompanies each article in each language. We ask the question “Was this article helpful?”, which can be answered with yes or no. Of course this metric is not perfect, articles that describe features have higher ratings than articles explaining how to fix a problem, English articles are generally higher rated than localized articles, despite having the same content, and the rating is also influenced by the path people took to get to the page. However, in this case we are not interested in the absolute ratings, we are particularly interested in the change since we moved to the new iA and design.

So, what happened?

We knew from previous tests that making the site browsable would be beneficial for that segment of our users who would rather browse than search for their article. People rate an article down, when it’s not the one that they were looking for. We know so much from our article surveys, and assuming we did our homework we should help more people find the right article. That being the case we expected the helpfulness of articles to rise, but it was hard to tell by how much it would rise. Considering that we have over 500,000 visitors per day and 80% use the KB, even a change by one percentage point would help an additional 1.46 million visitors per year. Without further ado, here are the results:

The results are phenomenal, we raised the helpfulness by 10 percent on average. That’s an increase by 5 percentage points and means an additional 7.3 million visitors per year stating that they found a SUMO article helpful. This is across all languages and across all incoming channels. It means that in 7.3 million cases where people might have decided to drop Firefox or be miserable because they couldn’t get Firefox to do what they wanted, they will now leave SUMO satisfied with their browser.

It’s hard to overstate the significance of this, and we are extremely happy with the results. The improvements to the site were the result of month of hard work by many people on the SUMO team, from SUMOdev, our creative team, and UX designers. We knew we were able to offer our users a better service, and the work has finally paid off. Continually thinking about how to serve our users better is what’s driving this team, and we will take these results as motivation to work even harder on improving our services.

Today, I’m very proud of what this team made possible, and I’d like to extend my thanks to each and everyone involved in the process: You made these results possible!

SUMO development in 2012

We are more than halfway through 2012 and this seems like a good time to stop and take a look back at the year so far. In particular I’d like to provide insights into how we used the available development time and what we were able to achieve for the community this year.

Search and Browsing on SUMO

Late in 2011 we had found out that our site needed some serious improvement in it’s search and information architecture. We tackled our search issues right away, and with the help of our UX designers and Susan, our information architecture expert, we spent the first 3 month of this year to come up with a site structure that would not only speak to our current users, but would also be flexible enough for future products. And while we implemented some of the resulting suggestions right away, we then took another quarter to test the new information architecture rigorously in the real world, before we wrote the first line of code. I’ll have more to say about that in a future blog post, but I wanted to take the time today to look at our development efforts for this year.

In very broad strokes, we spend the biggest part of our development time on moving to a new search engine and tuning that engine to return much better results than before. We also spent a large chunk of our time on measuring and displaying our key performance indicators (KPI), the metrics we take to measure our success as SUMO. More recently we have focused the majority of our effort on implementing the proposed new information architecture and a new “One Mozilla” design that will bring us visually closer to the existing Mozilla ecosystem.

Where did all the time go?

For the first time this year, we also tracked what user roles we served with the development time we had. The following table charts the number of points we spent, each point stands roughly for one day of development. As you can see the total number of points varies throughout the year as the constellation of the development team changes. We separated the user roles into “contributor”, “user”, “sumo-team”, and “dev”, which is used for infrastructure work. That is not to say that the classification was always clear cut. For example a lot of the development time  spend on contributors was also beneficial for the sumo team and the other way around. You can see all individual sprints on Scrumbugs, if you want to have a closer look.

 

Legend:

  • Contributor: Code changes that affected contributor of the site the most
  • User: Code changes that affected visitors of the site the most
  • SUMO Team: Code changes that affected SUMO staff the most
  • Development: Infrastructure work that didn’t benefit any one role in particular.

Community Focus

As is clear from the data we put our focus on the user experience this year. However we also invested heavily into the experience for the most important aspect of SUMO: contributors. I don’t want to list the dozens of individual bugs that were fixed and smaller feature that were added as you can quite easily see each of our sprints on Srumbugs. but I wanted to take the time to mention some of them as a reminder of what we worked on for individual contributors this year.

Forum contributors

  • Forum contributors can now add links to KB articles easily by searching for them in the “add link” overlay, which removes the need to open a new window, doing the search and linking manually.
  • We are now hiding questions that are older than 90 days and don’t have any answers from most parts of the site, as well as hiding all threads that are older than 180 days, so that we can focus on the threads in the forum that need our attention the most
  • We added a feedback indicator to the forum, that now tells us how far we are away from our goal to answer all incoming questions within 72 hours.
  • Forum contributors can now use the magic hat, that will offer them various small snippets that they can use to ask for more information, or answer recurring questions with 3 simple clicks.
  • Also, we are now only bumping questions to the top when there is a reply and ignoring other activity, to make sure we can focus on the most important questions.

KB contributors:

  • KB contributors can now send messages to all recent contributors of an article, even if they approve their own edit.
  • We added a remaining characters counter to the search summary. Google only shows the first 160 characters of a site as the preview, and after deciding to make that the search results summary we also added a counter to make sure we stay within the limits.

Localizers

  • The “approved” mail notification now mentions the approver and the changes in the body, so that people can save a round trip to SUMO.
  • In the same way, the “ready for localization” email now features a diff, so contributors can tell from their inbox how much effort a new update would take.
  • We are now showing messages above English articles, if the visitor is coming from a localized article, and we ask people to help us with the translation of that article into the language of the visitor.
  • Localizers can now see all navigation articles on their L10n dashboard, which saves us searching for it on various documentation pages.

Army of Awesome:

  • The Army of Awesome snippets can now be easily edited on SUMO, since they are KB articles now, instead of being in code.
  • The Army of Awesome has now statistics that more accurately display the effort of our community and our goals.
  • SUMO members can now stayed signed in on the Army of Awesome and answer tweets without having to log-in to Twitter each time.

All:

  • We have added a simple quoting feature for private messages, that now makes it possible to cite text in a reply and giving context to a reply.
  • We have adjusted several time frames on the KB dashboards to be able to react to changes more quickly
  • Most parts of SUMO now feature time stamps that consider the time zone of the user instead of being fixed to Pacific Time
  • Support forum and discussion forum posts of users are now displayed on the profile pages of all users for quick reference.

What’s next?

All in all I’m very proud of what we all have done so far and I can’t wait to see in production the many changes that are planned for this year, the most important of which is currently under development: The new information architecture and redesign. The new iA has been on our minds for most of this year already offers tremendous wins for users and contributors alike, and I’ll soon blog about the project and its expected effect on our site. For now, a big thank you goes out to all sumo developers, contributors and members. You are what makes SUMO so awesome!

The 8th SUMO sprint

Last Tuesday we finished the 8th SUMO sprint of this year. I’s been a short sprint, because for half of it Mozilla web developers met for a work week in California. Nevertheless we were able to land some great improvements to SUMO.

  1. One of our top priorities right now is improving our search results. To that end we started an initiative to rate KB and forum results as equals, based solely on the content instead of showing KB articles on top and forum posts at the bottom. We hope that this step will result in better search results by higher click through rates. This is a project that will take more than one sprint to be implemented. We completed the first of 3 phases in this sprint and are planning to implement the rest over the next sprints. You can read Will’s bug comment about it, if you want to know more about the gory details of the unification work.
  2. We were also able to implement a feature in the forums that gives our community an indicator for how we are doing and shows everyones impact on the forum: We call it the forum feedback indicator. It shows you how many questions were asked in the last 3 days and how many have a reply. It also links to the questions that haven’t been answered yet. The progress bar is orange, for any values below 100% and turns green when all questions are taken care of.

As with every sprint we also fixed a number of annoying bugs, and you can always follow along on the excellent scrumbugs. Here is the list for the 8th sprint.

However I’d like to point out one specific issue, that was bugging us in the forums for quite a while, creeping back when we thought we had squashed it: In some cases, a question displayed “No replies”, even though it already had replies. That bug is now hopefully fixed for good.

Meet Brinda!

Say hi to Brinda! She’s an intern for SUMO this summer, working on text mining support questions to find our top issues. Automating this manual process will be a huge help to us. I shot this video almost a month ago so Brinda is wrapping up her work now and will be heading back to the University of Illinois soon for her senior year. It’s been great having her on the SUMO team.

Meet Michelle!

Please join me in welcoming Michelle Luna to Mozilla! Michelle will be heading up our mobile Firefox support efforts and ensuring that we provide world class, scalable, self-service and community-driven support for mobile Firefox users.

We’re very excited that Michelle has joined our Army of Awesome because she’s pretty darn awesome herself — she has a ton of experience with customer support, community management, content writing, and working closely with development and QA teams. This will all be a great help for us in our efforts to turn support into something that integrates closely with product planning in addition to just being a great service for our users.

Michelle graduated from the University of Colorado in 1999 with a BA in Technical Communications, an enthusiasm for software support and a passion for playing guitar and snowboarding. In 2002 she moved to San Francisco to join Sun Microsystems where she led the release of Solaris documentation as open source and went on to chair the OpenSolaris Governing Board 2009/2010! Most recently, Michelle served as principle program manager in support of the ZFS Storage appliance at Oracle, Inc.

Meet Ibai!


Please join me in welcoming Ibai Garcia to Mozilla! Ibai (in some circles also known as El Fuego de la Cueva) is the SUMO team’s new Services Support Coordinator. He will lead the services support effort, including redefining the experience on multiple platforms, analyzing metrics and user feedback, and reporting metrics and support insights back to the product, engineering and QA teams.

Ibai has an impressive background in both online support and community management. In his most recent gig he was the Online Customer Service Manager at Vodafone in Spain, where he successfully scaled their support by integrating user-to-user help with the more traditional customer support channels (e-mail, phone, help center).

We’re all super excited to have Ibai joining the SUMO team.

P.S. You can follow Ibai on Twitter and read his blog (if you read Spanish).

about:sumo – May 2009

NOTE: This is the blog version of our about:SUMO newsletter. To subscribe to the email version, sign up using the online subscription form.

about:SUMO

Welcome to about:SUMO — a digest of news from Mozilla’s support world! With this being the first issue, don’t hesitate to send us your feedback on about:SUMO and how we can improve it. Lately, we’ve had a few really important additions, such as the localization dashboard and a new search engine; and next month will be just as good.

Localization Dashboard

In SUMO 1.0, we implemented a dashboard for localizers, so they can get an overview of which pages are most important to translate. It also tracks how much of the knowledge base has been translated. You can view the localization dashboard now!

We’ve switched search engines

One of the new features that was launched in the last SUMO update is a new search engine. By using our own search engine, we can tailor results based on how useful pages are as well as return KB results and forum results together. This not only saves users time, but helps them get the best support in fewer steps. Furthermore, the data from these searches can help us more quickly pinpoint common issues and improve our support resources. You can read more in this blog post.

Get ready for screencasts

We’ve put a lot of effort into adding multimedia to support. The ability to show a person what to do in addition to telling them, makes instructions a lot easier to understand; and some of our most positive knowledge base feedback has been about the existence of screenshots. Soon, we’ll be implementing the ability to add videos (a.k.a. screencasts) to knowledge base articles. We will support both Flash and the open video format called Ogg/Theora, which the upcoming Firefox 3.5 will support natively (without the need of a plugin). For more information, read our recent blog post.

Get notified whenever a page in your language is created or edited

Keeping track of changes to the knowledge base is essential for keeping it up to date; and being notified of changes makes it convenient. In addition to monitoring individual articles, we’ve implemented some tools to make it possible to monitor all knowledge base articles for a specific language, which is something localizers have been requesting. In addition, you can monitor all articles in the “How to contribute” category, or even articles marked for Firefox 3.5. Give it a try.

Status update on the knowledge base update for Firefox 3.5

There is no better time to contribute than now! Firefox 3.5 is going to be released soon, and the entire knowledge base needs to be updated. It’s a giant opportunity to help and get to know Firefox 3.5, so you’ll be able to answer user questions when Firefox 3.5 is released. To get involved, read our Firefox 3.5 support status update.

Looking at writing more concise documents

After website usability expert Neil Lee from the webdev team made a usability audit of the Firefox Support website, there has been a growing movement to make the documentation more concise. By making knowledge base articles easier to read, you can make it helpful to users who might otherwise be less patient reading long-winded instructions.

Surveys for Live Chat and Forum contributors

We’re constantly working to improve SUMO for our contributors as well as our users. As such we’ve sent out surveys to current and recent contributors to get your feedback on how we can make it even more fun and easy to help our users. We really want to hear from you and your comments will help make SUMO even better for everyone. If you didn’t get a copy of the survey, let us know at feedback[at]support.mozilla.com and we’ll e-mail you the link. Stay tuned next month as we go over the results.

Spend 10 minutes a day doing support

One of the quickest and easiest ways to help out Firefox users on SUMO is to answer forum questions. You can do it for as long as you like and whenever you like and it takes just minutes to help out a user. Try helping out for just ten minutes a day and see how easy and fun it is.

New and ongoing support issues

In this section (which will be in every about:sumo newsletter) we’ll be covering one common issue on SUMO for the past month.  This month, one of the top issues was sqlite corruption/locking in the profile folder.  Firefox 3.0 introduced the sqlite database format for places (bookmarks and history) and cookies using places.sqlite and cookies.sqlite respectively.  Unfortunately, these files can sometimes corrupt if Firefox or the computer is shut down improperly.

Another common cause is security software (AVG in particular) detecting tracking cookies or malicious sites in places and locking/corrupting the file when they try to clean them.  When this happens, Firefox can crash on startup, lose or not save bookmarks/history/cookies, have a malfunctioning location bar or return random characters in cookies or bookmark searchs.

Firefox 3.0.11 will have an updated SQLite engine and notifications that will be more resistant to this and alert users.  In the meantime, the steps detailed here for places.sqlite and here for cookies.sqlite will help address this. Watch out for this issue in the forums and on Live Chat and help these users with these steps.

The about:sumo experiment

A while back, the idea of creating an about:sumo newsletter was
proposed. Since then, we’ve been so busy with other parts of SUMO, that
we never got to follow through on that idea.

We haven’t forgotten! In the next three months, we’d like to test out
the idea, and see how it goes. The purpose of the newsletter would be to
provide a digest of news from the SUMO world for those who are not
actively involved, but still interested in knowing what we’re working on.

Over the next three months, we’ll publish a newsletter each month, and
depending on the feedback from the community, we’ll know whether or not
to make the newsletter a regular thing. Stay tuned for more information.