Category Archives: Development

SUMO now helping an additional 7.3 million visitors

About 3 weeks ago we made the switch to a new information architecture and new design. The goal was to improve the browsability of the site and help people find the articles that they were looking for. 3 Weeks later we can now take a look at our key performance indicators to determine whether the whole project was worth the effort

Methodology

Since this project was primarily concerned with the Knowledge Base, we can focus on the helpfulness rating in this channel. Also, we know from our exit surveys that about 80% of our visitors use the KB.  The KB helpfulness rating is based on the survey that accompanies each article in each language. We ask the question “Was this article helpful?”, which can be answered with yes or no. Of course this metric is not perfect, articles that describe features have higher ratings than articles explaining how to fix a problem, English articles are generally higher rated than localized articles, despite having the same content, and the rating is also influenced by the path people took to get to the page. However, in this case we are not interested in the absolute ratings, we are particularly interested in the change since we moved to the new iA and design.

So, what happened?

We knew from previous tests that making the site browsable would be beneficial for that segment of our users who would rather browse than search for their article. People rate an article down, when it’s not the one that they were looking for. We know so much from our article surveys, and assuming we did our homework we should help more people find the right article. That being the case we expected the helpfulness of articles to rise, but it was hard to tell by how much it would rise. Considering that we have over 500,000 visitors per day and 80% use the KB, even a change by one percentage point would help an additional 1.46 million visitors per year. Without further ado, here are the results:

The results are phenomenal, we raised the helpfulness by 10 percent on average. That’s an increase by 5 percentage points and means an additional 7.3 million visitors per year stating that they found a SUMO article helpful. This is across all languages and across all incoming channels. It means that in 7.3 million cases where people might have decided to drop Firefox or be miserable because they couldn’t get Firefox to do what they wanted, they will now leave SUMO satisfied with their browser.

It’s hard to overstate the significance of this, and we are extremely happy with the results. The improvements to the site were the result of month of hard work by many people on the SUMO team, from SUMOdev, our creative team, and UX designers. We knew we were able to offer our users a better service, and the work has finally paid off. Continually thinking about how to serve our users better is what’s driving this team, and we will take these results as motivation to work even harder on improving our services.

Today, I’m very proud of what this team made possible, and I’d like to extend my thanks to each and everyone involved in the process: You made these results possible!

The all new SUMO

Today we are going to make one of the biggest changes yet to SUMO, the Mozilla Support site, and this blog post is about what changes we are making. The changes will effect you the most as a user, but there are a number of changes for contributors as well.

First, a little history, what’s the SUMO team been up to this year?

For the last 9 month the SUMO team has been working on a new way to let users access our site content. Until recently the only way to reach most of our articles was by search, or by following links in articles. This is how wikis traditionally work. Of course that way of accessing content only works for a part of our visitors, some people want to search and some people want to browse to the solution, drilling down with ever finer topics to reduce the number of article that are related to the issue.

To come up with a new information architecture that would let people drill down like that, we first researched the mental model of our site users, how they think about issues and in what categories they would look for them. Based on that we created a small number of base categories and assigned our articles to those categories.

The next step was figuring out how to make this information architecture visible. We started to lay out a number of alternatives on paper and tested with real people in a lab. This paper prototyping gave us a way to test a number of ways to lay out the information very quickly. After a number of iterations we settled on the final designs and workflows.

Now we had everything to start adapting our site, but since this would be a big redesign, and we’d soon switch to the new unified One Mozilla design anyway, the decision was made to use this opportunity to rebuild the site based on the new theme, and that’s why the changes today not only affect the KB, but every part of our site.

So, what is changing? What does it look like?

The main change is, that we now support several products from one start page and all articles can be accessed by browsing. Let’s start with the start page:

We have the main topics on top, they allow you to start browsing by selecting your issue first, and then the product you have issue with.

One step below you can see the hot topics. Those are actually articles, things that came up recently and affect a large number of people. By providing them upfront we save a large number of people the hassle of searching or browsing for their solution.

Below that we have the product picker, this is a way to navigate our content by choosing the product first and then narrowing down the topics.

No matter what way you select, topic first or product first, you’ll end up narrowing down the number of articles to a scanable few and proceed to read one of the articles.

The important thing for localizers to note is: all of this is automated, there is no need anymore to create navigation pages and all the confusion that brought with it.

So, how did the article view change? On the surface not much has changed, but because we keep track of topics, we can now offer you a way to move to related topics, which is particularly interesting for people landing on articles from external searches:

Much, much more has changed, but this is the gist for the KB part of the site.

So, what has changed for forum contributors?

While the new iA did not touch the support forum per se, we took the redesign as an opportunity to improve a number of factors in the listing of questions for our contributors.

The new design is more friendly and clean, but at the same time gives more information about the thread content at the same time. This is especially helpful when contributors scan the thread listing page deciding which thread to pick next.

We already started rolling the design out to our contributor base over the last week and will start rolling it out to 1% of the general audience today. If everything goes to plan we’ll make it available to the general audience on Monday. If you want to try it out now, just register an account, and if you have any feedback, please use the comment section below.

The new information architecture will open up our content to a whole new group of users and make it much more accessible, while our new design is more coherent, taking into account all of the features we added since our first release while also being consistent with the Mozilla sites in general. All of this makes us very excited and hopeful that we’ll get that much closer to our number one goal: Happy users!

SUMO development in 2012

We are more than halfway through 2012 and this seems like a good time to stop and take a look back at the year so far. In particular I’d like to provide insights into how we used the available development time and what we were able to achieve for the community this year.

Search and Browsing on SUMO

Late in 2011 we had found out that our site needed some serious improvement in it’s search and information architecture. We tackled our search issues right away, and with the help of our UX designers and Susan, our information architecture expert, we spent the first 3 month of this year to come up with a site structure that would not only speak to our current users, but would also be flexible enough for future products. And while we implemented some of the resulting suggestions right away, we then took another quarter to test the new information architecture rigorously in the real world, before we wrote the first line of code. I’ll have more to say about that in a future blog post, but I wanted to take the time today to look at our development efforts for this year.

In very broad strokes, we spend the biggest part of our development time on moving to a new search engine and tuning that engine to return much better results than before. We also spent a large chunk of our time on measuring and displaying our key performance indicators (KPI), the metrics we take to measure our success as SUMO. More recently we have focused the majority of our effort on implementing the proposed new information architecture and a new “One Mozilla” design that will bring us visually closer to the existing Mozilla ecosystem.

Where did all the time go?

For the first time this year, we also tracked what user roles we served with the development time we had. The following table charts the number of points we spent, each point stands roughly for one day of development. As you can see the total number of points varies throughout the year as the constellation of the development team changes. We separated the user roles into “contributor”, “user”, “sumo-team”, and “dev”, which is used for infrastructure work. That is not to say that the classification was always clear cut. For example a lot of the development time  spend on contributors was also beneficial for the sumo team and the other way around. You can see all individual sprints on Scrumbugs, if you want to have a closer look.

 

Legend:

  • Contributor: Code changes that affected contributor of the site the most
  • User: Code changes that affected visitors of the site the most
  • SUMO Team: Code changes that affected SUMO staff the most
  • Development: Infrastructure work that didn’t benefit any one role in particular.

Community Focus

As is clear from the data we put our focus on the user experience this year. However we also invested heavily into the experience for the most important aspect of SUMO: contributors. I don’t want to list the dozens of individual bugs that were fixed and smaller feature that were added as you can quite easily see each of our sprints on Srumbugs. but I wanted to take the time to mention some of them as a reminder of what we worked on for individual contributors this year.

Forum contributors

  • Forum contributors can now add links to KB articles easily by searching for them in the “add link” overlay, which removes the need to open a new window, doing the search and linking manually.
  • We are now hiding questions that are older than 90 days and don’t have any answers from most parts of the site, as well as hiding all threads that are older than 180 days, so that we can focus on the threads in the forum that need our attention the most
  • We added a feedback indicator to the forum, that now tells us how far we are away from our goal to answer all incoming questions within 72 hours.
  • Forum contributors can now use the magic hat, that will offer them various small snippets that they can use to ask for more information, or answer recurring questions with 3 simple clicks.
  • Also, we are now only bumping questions to the top when there is a reply and ignoring other activity, to make sure we can focus on the most important questions.

KB contributors:

  • KB contributors can now send messages to all recent contributors of an article, even if they approve their own edit.
  • We added a remaining characters counter to the search summary. Google only shows the first 160 characters of a site as the preview, and after deciding to make that the search results summary we also added a counter to make sure we stay within the limits.

Localizers

  • The “approved” mail notification now mentions the approver and the changes in the body, so that people can save a round trip to SUMO.
  • In the same way, the “ready for localization” email now features a diff, so contributors can tell from their inbox how much effort a new update would take.
  • We are now showing messages above English articles, if the visitor is coming from a localized article, and we ask people to help us with the translation of that article into the language of the visitor.
  • Localizers can now see all navigation articles on their L10n dashboard, which saves us searching for it on various documentation pages.

Army of Awesome:

  • The Army of Awesome snippets can now be easily edited on SUMO, since they are KB articles now, instead of being in code.
  • The Army of Awesome has now statistics that more accurately display the effort of our community and our goals.
  • SUMO members can now stayed signed in on the Army of Awesome and answer tweets without having to log-in to Twitter each time.

All:

  • We have added a simple quoting feature for private messages, that now makes it possible to cite text in a reply and giving context to a reply.
  • We have adjusted several time frames on the KB dashboards to be able to react to changes more quickly
  • Most parts of SUMO now feature time stamps that consider the time zone of the user instead of being fixed to Pacific Time
  • Support forum and discussion forum posts of users are now displayed on the profile pages of all users for quick reference.

What’s next?

All in all I’m very proud of what we all have done so far and I can’t wait to see in production the many changes that are planned for this year, the most important of which is currently under development: The new information architecture and redesign. The new iA has been on our minds for most of this year already offers tremendous wins for users and contributors alike, and I’ll soon blog about the project and its expected effect on our site. For now, a big thank you goes out to all sumo developers, contributors and members. You are what makes SUMO so awesome!

The new magic SUMO tool for frequent questions, and an interview with its developer, Tobias Markus

For a few weeks now we have been replying to almost every single question that is asked in the Mozilla support forums. That is up from about 50% last year and it shows the dedication of our community. Also over the last few month we have made a number of changes to SUMO to reduce the flow of simple questions that have answers in the KB already. That has been quite succesful, not only are we getting really hard questions now, but they are solved at a higher rate than before too.

The challenge now is that we have to ask for more information before we can solve an issue. Repeating those questions is quite tiring, so we decided to build a tool to help out our community and gather all the important questions to ask in one place.

We had actually planned to work on it later this year, but one of our great community members, Tobbi, saw an opportunity and offered to try his hand on it.

The result went online a few days ago, and is amazing. Sign into SUMO, and check it out for yourself:

  1. Chose a question
  2. Click on the response tool (the magic hat symbol)
  3. Select an answer or a request for specific information
  4. Customize as necessary and send!

Over time this tool will hopefully gather all the shared wisdom of our community and help us solve issues in as few steps as possible.

Interview with the developer, Tobias Markus.

This is not the first time a community member has been helpful in the development of Kitsune, the software that powers SUMO, but it’s the first major feature. We’d love to see much more of this, and I asked Tobbi for an interview to ask him how he got involved for anyone who wants to follow in his footsteps.

Tobbi, you have been a SUMO community member for quite a long time. Where did you start and how was your SUMO journey so far?

It was late 2008 when I started contributing to SUMO. My portal to the SUMO world was the Firefox LiveChat, which I joined as I really liked the personal touch that the chat had. Later on, I found out that there was a German localization group for SUMO articles, so I started translating articles into German and helped reaching the goal of then 100% localization coverage. I even gave forum support, although I preferred LiveChat because it was more personal than the forums.

How did you get the idea to help with the development of SUMO?

The idea for development started when I was working on a Firefox add-on together with Propeng. We implemented various features for the old forums to make them more contributor-friendly and add some tools that aimed at making life for contributors easier. However, due to the amount of unfinished code and the switch to Kitsune, the project was stalled. That’s when we decided to split the big project into smaller chunks to make them compatible again with the new forums. One of these smaller projects was the Knowledge Base article autocomplete that was landed a few months ago. I think it was Rosana who suggested integrating the Knowledge Base autocomplete into the Firefox support website directly. That’s when I started contributing to the development of SUMO.

Did you get any help with your project?

The SUMO team and other contributors really helped me a lot. At first I was posting an early draft to the contributors forum to gather suggestions from other contributors and the team. The feedback I got was huge. I spent the next couple of weeks reacting on it and making a lot of improvements to the code to enable things like instant preview, edit mode, and even flags that can limit certain responses to certain groups of contributors. The SUMOdev team also answered some implementation questions.

Once this was done, the next thing was to make the experience perfect for contributors. I brainstormed a lot with Bram [the SUMO UX designer] about best practices in order to get a good user experience design and spent some more weeks making the necessary changes.

The fourth step was a final code review by Ricky Rosario. That went smoothly as well, the necessary changes I had to do were minimal and most of them were about simple code style fixes. Once these were fixed, the code was ready to land.

The final step was to create the canned responses. I asked the community again to get back to me with some frequently used responses and the amount of responses I got was overwhelming. I chose around 10 of the most frequently used ones (based on my own judgement) and rewrote them a little. Kadir then helped me put each response in its own article and create a main article for the category structure that has links to all the individual responses.

What is your advise for people who want to get involved with SUMO development?

Drop by in #sumodev on irc.mozilla.org and say hello. The SUMOdev people answered any questions I had so that canned responses could become a success. Also, there is good documentation available online about setting up a local copy of SUMO. So, if you have a great feature in mind for the support website and want to start developing, give it a go!

Tobi, thank you for this interview, and in the name of the SUMO community: thanks for making everyone’s live so much easier!

The 8th SUMO sprint

Last Tuesday we finished the 8th SUMO sprint of this year. I’s been a short sprint, because for half of it Mozilla web developers met for a work week in California. Nevertheless we were able to land some great improvements to SUMO.

  1. One of our top priorities right now is improving our search results. To that end we started an initiative to rate KB and forum results as equals, based solely on the content instead of showing KB articles on top and forum posts at the bottom. We hope that this step will result in better search results by higher click through rates. This is a project that will take more than one sprint to be implemented. We completed the first of 3 phases in this sprint and are planning to implement the rest over the next sprints. You can read Will’s bug comment about it, if you want to know more about the gory details of the unification work.
  2. We were also able to implement a feature in the forums that gives our community an indicator for how we are doing and shows everyones impact on the forum: We call it the forum feedback indicator. It shows you how many questions were asked in the last 3 days and how many have a reply. It also links to the questions that haven’t been answered yet. The progress bar is orange, for any values below 100% and turns green when all questions are taken care of.

As with every sprint we also fixed a number of annoying bugs, and you can always follow along on the excellent scrumbugs. Here is the list for the 8th sprint.

However I’d like to point out one specific issue, that was bugging us in the forums for quite a while, creeping back when we thought we had squashed it: In some cases, a question displayed “No replies”, even though it already had replies. That bug is now hopefully fixed for good.

SUMO sprint 2012.4

This is a quick update on what we are currently working on in the 4th SUMO sprint of this year, the results of this sprint will go live between February 28th and March 6th.

Our dev goals for this quarter were to get the SUMO KPI dashboard up and running, and to switch to a new search engine that would allow us to implement all the improvements suggested by the ongoing taxonomy/ux research and the usability research before that.

We have recently turned our new search engine (Elastic Search) on for 50% of our site’s users, and the early results are very promising. The Click Through Rate, our main success metric at this point, of the new engine is already better than that of the old one. Currently we are gathering more data to get reliable results, but if the final results agree with our preliminary findings, we’ll switch to Elastic Search 100% in early March and start improving our results. One big issue we’ll tackle soon is the performance of Elastic Search, currently it is much slower than our old search engine. However Will already has ideas that will hopefully give us significant improvements in this area, starting with sprint 5.

Our KPI dashboard is nearing completion, thanks to Ricky. We already have many of the metrics we set out to present and few are left. This sprint will give us the Army of Awesome numbers, the number of unique visitors we have at SUMO and our localization coverage.

One change that seems small, but will hopefully have a big effect, is the change of our font, colors and sizes. We had numerous complaints that our text was hard to read and links were easily overlooked. Bram, our UX designer, and font expert extraordinaire, came up with a new style that uses stronger colors for links, changes the gray on white of our body text to a darker black, and moves us from serif to sans-serif. The latter change is significant, there is a large body of research on what is better for readability serif, or sans-serif, and amid a lot of myths there are arguments for both sides. However it seems clear that there is only one factor that reliably affects readability: familiarity. Of the top 100 websites today only 4 use a serif font for body text. Since users today are so much more familiar with sans-serif our hope is that this change will contribute to the overall readability of SUMO.

Another thing we have been working on is several customizations of our contributor forums, so they could be used by the Firefox Aurora and Beta communities. Expect more activity in those forums very soon.

As always, there is a good number of new features and fixes for smaller bugs:

ID Summary
730065 Allow sort by original post date
718813 let people add screenshots when asking a question
726037 Combine some of the KPI dashboard graphs
645546 How to contribute articles overview in the dashboard
724833 Change AoA signposts
625841 Reword email notifications for Questions
720226 We need reviewing rights for en-US and other locales
724483 remove en-US part from link in ‘ready for localization’ emails
725287 Use One Mozilla breadcrumb style (revert bug 716018)
726112 Allow entering a message for recent contributors, even if you approve your own revision.
722697 Show contributor sidebar in the questions app
726856
706948 In some cases, a question displays “No replies” with a reply below
722509 notification about private message has localized URL
728307 Please add {for} support for fx12, m12 and add Firefox 12 to the dropdown UI, remove Firefox 9 from the UI

SUMO is adding new features: group dashboards and private messaging

Hello everyone,

Starting today we will be beta testing the new groups and private messaging features of Kitsune. We want to start with a small number of people for now, so we are going to cap this at about ten. We will have a test day for everyone on July 8th. If you are interested in testing private messaging and giving feedback on it, please let me know in this thread on the support community forum and I’ll activate it for you.

If you are a localizer, you can also request the group dashboard feature. This will add another tab to your dashboard and those of your team members. That tab will hold your localization dashboard and a message on top that you as the locale leader can change to message your team. Also, your localization group will get a profile page listing the locale leader and every member of the team.

You can see here what the German localization group profile looks like.

For the groups dashboard I wrote a short tutorial that you can see here.

Private messaging should be fairly intuitive. Once the feature is activated, just click on “Inbox” on the upper right corner of the page. Make sure that you only message people who are listed in the forum thread for now.

Please report any issues that you encounter on this etherpad so we can fix them quickly.

And of course, please let me know if you have any questions.

Thanks!
Kadir

SUMO 2.4 – The end of the tunnel

Two weeks ago, we released SUMO 2.4, completing a year-long project to replace the SUMO platform!

SUMO 2.4 moved the last bits of functionality into Kitsune, our Django-based platform. These include user features like login/logout, registration, and profiles.

This represents a significant milestone and success for the SUMO project, and is particularly meaningful to the development team. We’ve been working toward this since January 2010, and seeing it completed is an amazing feeling.

Over the past year we’ve progressively replaced pieces of our old platform with new code:

  • In May, we took our first step by transitioning to new Search Result Pages.
  • In July, we switched the Discussion Forums, and started authenticating users in both systems.
  • In August, we turned on the new Support Forum section.
  • In September, we added the new Army of Awesome, built very rapidly on the new platform.
  • Just recently, in November, we brought the new Knowledge Base, the largest, and most complex part of SUMO, online.
  • And with 2.4, we’ve brought over the last piece, User Accounts.

This final step in the migration to Kitsune opens up a bunch of new doors for features and improvements. For example, user registration is much simpler now. We’re transitioning data internally to be more secure. The entire site is faster and puts a lot less load on our servers, meaning we can serve more traffic with the same hardware.

We are especially happy we were able to complete this transition before the Firefox 4 release. Being entirely on the new platform gives us more confidence in our ability to keep helping users even with traffic spikes from the release.

We devoted 2010 to investing in this new platform, designed specifically to make it easier for our awesome community to help 400 million Firefox users worldwide. In 2011, we’ll start seeing the payoff of that investment, for our developers, contributors, and users, and expect to see SUMO really take off!

What’s new in Kitsune

Ludicrous Speed
In my last post I wrote about the process of creating our new Knowledge Base but that did not answer the question of how the new KB will improve the experience for users and contributors.

There are so many exciting parts to the new Knowledge Base that it’s hard to know what to talk about first. The biggest improvement was also the most requested – speed. With our old KB it was often hard to find out why pages timed out and even harder to actually do anything about it. With the new system we know every detail and can tweak the performance, move things around and make sure that users will see a page in an instant. As has been shown time and time again, speed is the ultimate measure of usefulness. Even a 3 second response time will often lead to users leaving the site or being unsatisfied with the interaction no matter how good the content. So we’ve focused heavily on this.

The latest numbers we have show a huge improvement in response times – to be precise, a 28-fold increase in handled requests per second! That’s incredible indeed. The best part is that it will make the KB more useful for users as well as contributors. Gone are the days of timed out dashboards and slow loading pages.

What you can do!

Of course speed is not the only improvement with this new system. Over the next few weeks leading up to the release, we will write more about specific new features and improvements. But you don’t have to wait for the November 30th release to see for yourself. Take a look at the work-in-progress on our staging server.

If you are interested, help us make it as bug free as possible by joining us for a QA day on November 12th. From 8am to 5pm PST we’ll gather in #sumo on IRC to make sure the KB is ready for the release. We’d love to see you there!

SUMO 2.1: New Discussion Forums

On Wednesday this week, we migrated the discussion forums on support.mozilla.com to our new platform.

The forums we moved over are

We did not move the Firefox support forum: that is coming in our next major milestone. These discussion forums join search results as the second component of the site to move over to the new platform.

The new discussion forums are very similar to the old discussion forums. We tried to keep big changes to a minimum. If you see something that doesn’t seem like it’s working right, let us know in the comments here, or in the Contributors’ forum.