Category Archives: Knowledge Base

SUMO now helping an additional 7.3 million visitors

About 3 weeks ago we made the switch to a new information architecture and new design. The goal was to improve the browsability of the site and help people find the articles that they were looking for. 3 Weeks later we can now take a look at our key performance indicators to determine whether the whole project was worth the effort

Methodology

Since this project was primarily concerned with the Knowledge Base, we can focus on the helpfulness rating in this channel. Also, we know from our exit surveys that about 80% of our visitors use the KB.  The KB helpfulness rating is based on the survey that accompanies each article in each language. We ask the question “Was this article helpful?”, which can be answered with yes or no. Of course this metric is not perfect, articles that describe features have higher ratings than articles explaining how to fix a problem, English articles are generally higher rated than localized articles, despite having the same content, and the rating is also influenced by the path people took to get to the page. However, in this case we are not interested in the absolute ratings, we are particularly interested in the change since we moved to the new iA and design.

So, what happened?

We knew from previous tests that making the site browsable would be beneficial for that segment of our users who would rather browse than search for their article. People rate an article down, when it’s not the one that they were looking for. We know so much from our article surveys, and assuming we did our homework we should help more people find the right article. That being the case we expected the helpfulness of articles to rise, but it was hard to tell by how much it would rise. Considering that we have over 500,000 visitors per day and 80% use the KB, even a change by one percentage point would help an additional 1.46 million visitors per year. Without further ado, here are the results:

The results are phenomenal, we raised the helpfulness by 10 percent on average. That’s an increase by 5 percentage points and means an additional 7.3 million visitors per year stating that they found a SUMO article helpful. This is across all languages and across all incoming channels. It means that in 7.3 million cases where people might have decided to drop Firefox or be miserable because they couldn’t get Firefox to do what they wanted, they will now leave SUMO satisfied with their browser.

It’s hard to overstate the significance of this, and we are extremely happy with the results. The improvements to the site were the result of month of hard work by many people on the SUMO team, from SUMOdev, our creative team, and UX designers. We knew we were able to offer our users a better service, and the work has finally paid off. Continually thinking about how to serve our users better is what’s driving this team, and we will take these results as motivation to work even harder on improving our services.

Today, I’m very proud of what this team made possible, and I’d like to extend my thanks to each and everyone involved in the process: You made these results possible!

The all new SUMO

Today we are going to make one of the biggest changes yet to SUMO, the Mozilla Support site, and this blog post is about what changes we are making. The changes will effect you the most as a user, but there are a number of changes for contributors as well.

First, a little history, what’s the SUMO team been up to this year?

For the last 9 month the SUMO team has been working on a new way to let users access our site content. Until recently the only way to reach most of our articles was by search, or by following links in articles. This is how wikis traditionally work. Of course that way of accessing content only works for a part of our visitors, some people want to search and some people want to browse to the solution, drilling down with ever finer topics to reduce the number of article that are related to the issue.

To come up with a new information architecture that would let people drill down like that, we first researched the mental model of our site users, how they think about issues and in what categories they would look for them. Based on that we created a small number of base categories and assigned our articles to those categories.

The next step was figuring out how to make this information architecture visible. We started to lay out a number of alternatives on paper and tested with real people in a lab. This paper prototyping gave us a way to test a number of ways to lay out the information very quickly. After a number of iterations we settled on the final designs and workflows.

Now we had everything to start adapting our site, but since this would be a big redesign, and we’d soon switch to the new unified One Mozilla design anyway, the decision was made to use this opportunity to rebuild the site based on the new theme, and that’s why the changes today not only affect the KB, but every part of our site.

So, what is changing? What does it look like?

The main change is, that we now support several products from one start page and all articles can be accessed by browsing. Let’s start with the start page:

We have the main topics on top, they allow you to start browsing by selecting your issue first, and then the product you have issue with.

One step below you can see the hot topics. Those are actually articles, things that came up recently and affect a large number of people. By providing them upfront we save a large number of people the hassle of searching or browsing for their solution.

Below that we have the product picker, this is a way to navigate our content by choosing the product first and then narrowing down the topics.

No matter what way you select, topic first or product first, you’ll end up narrowing down the number of articles to a scanable few and proceed to read one of the articles.

The important thing for localizers to note is: all of this is automated, there is no need anymore to create navigation pages and all the confusion that brought with it.

So, how did the article view change? On the surface not much has changed, but because we keep track of topics, we can now offer you a way to move to related topics, which is particularly interesting for people landing on articles from external searches:

Much, much more has changed, but this is the gist for the KB part of the site.

So, what has changed for forum contributors?

While the new iA did not touch the support forum per se, we took the redesign as an opportunity to improve a number of factors in the listing of questions for our contributors.

The new design is more friendly and clean, but at the same time gives more information about the thread content at the same time. This is especially helpful when contributors scan the thread listing page deciding which thread to pick next.

We already started rolling the design out to our contributor base over the last week and will start rolling it out to 1% of the general audience today. If everything goes to plan we’ll make it available to the general audience on Monday. If you want to try it out now, just register an account, and if you have any feedback, please use the comment section below.

The new information architecture will open up our content to a whole new group of users and make it much more accessible, while our new design is more coherent, taking into account all of the features we added since our first release while also being consistent with the Mozilla sites in general. All of this makes us very excited and hopeful that we’ll get that much closer to our number one goal: Happy users!

The 8th SUMO sprint

Last Tuesday we finished the 8th SUMO sprint of this year. I’s been a short sprint, because for half of it Mozilla web developers met for a work week in California. Nevertheless we were able to land some great improvements to SUMO.

  1. One of our top priorities right now is improving our search results. To that end we started an initiative to rate KB and forum results as equals, based solely on the content instead of showing KB articles on top and forum posts at the bottom. We hope that this step will result in better search results by higher click through rates. This is a project that will take more than one sprint to be implemented. We completed the first of 3 phases in this sprint and are planning to implement the rest over the next sprints. You can read Will’s bug comment about it, if you want to know more about the gory details of the unification work.
  2. We were also able to implement a feature in the forums that gives our community an indicator for how we are doing and shows everyones impact on the forum: We call it the forum feedback indicator. It shows you how many questions were asked in the last 3 days and how many have a reply. It also links to the questions that haven’t been answered yet. The progress bar is orange, for any values below 100% and turns green when all questions are taken care of.

As with every sprint we also fixed a number of annoying bugs, and you can always follow along on the excellent scrumbugs. Here is the list for the 8th sprint.

However I’d like to point out one specific issue, that was bugging us in the forums for quite a while, creeping back when we thought we had squashed it: In some cases, a question displayed “No replies”, even though it already had replies. That bug is now hopefully fixed for good.

SUMO sprint 2012.4

This is a quick update on what we are currently working on in the 4th SUMO sprint of this year, the results of this sprint will go live between February 28th and March 6th.

Our dev goals for this quarter were to get the SUMO KPI dashboard up and running, and to switch to a new search engine that would allow us to implement all the improvements suggested by the ongoing taxonomy/ux research and the usability research before that.

We have recently turned our new search engine (Elastic Search) on for 50% of our site’s users, and the early results are very promising. The Click Through Rate, our main success metric at this point, of the new engine is already better than that of the old one. Currently we are gathering more data to get reliable results, but if the final results agree with our preliminary findings, we’ll switch to Elastic Search 100% in early March and start improving our results. One big issue we’ll tackle soon is the performance of Elastic Search, currently it is much slower than our old search engine. However Will already has ideas that will hopefully give us significant improvements in this area, starting with sprint 5.

Our KPI dashboard is nearing completion, thanks to Ricky. We already have many of the metrics we set out to present and few are left. This sprint will give us the Army of Awesome numbers, the number of unique visitors we have at SUMO and our localization coverage.

One change that seems small, but will hopefully have a big effect, is the change of our font, colors and sizes. We had numerous complaints that our text was hard to read and links were easily overlooked. Bram, our UX designer, and font expert extraordinaire, came up with a new style that uses stronger colors for links, changes the gray on white of our body text to a darker black, and moves us from serif to sans-serif. The latter change is significant, there is a large body of research on what is better for readability serif, or sans-serif, and amid a lot of myths there are arguments for both sides. However it seems clear that there is only one factor that reliably affects readability: familiarity. Of the top 100 websites today only 4 use a serif font for body text. Since users today are so much more familiar with sans-serif our hope is that this change will contribute to the overall readability of SUMO.

Another thing we have been working on is several customizations of our contributor forums, so they could be used by the Firefox Aurora and Beta communities. Expect more activity in those forums very soon.

As always, there is a good number of new features and fixes for smaller bugs:

ID Summary
730065 Allow sort by original post date
718813 let people add screenshots when asking a question
726037 Combine some of the KPI dashboard graphs
645546 How to contribute articles overview in the dashboard
724833 Change AoA signposts
625841 Reword email notifications for Questions
720226 We need reviewing rights for en-US and other locales
724483 remove en-US part from link in ‘ready for localization’ emails
725287 Use One Mozilla breadcrumb style (revert bug 716018)
726112 Allow entering a message for recent contributors, even if you approve your own revision.
722697 Show contributor sidebar in the questions app
726856
706948 In some cases, a question displays “No replies” with a reply below
722509 notification about private message has localized URL
728307 Please add {for} support for fx12, m12 and add Firefox 12 to the dropdown UI, remove Firefox 9 from the UI

Help us test private messaging and group dashboards today

Hey everyone,

This is a quick reminder about our test day for private messaging and group dashboards today, Friday, July 8th, from 3pm to 12pm UTC. We want to test the recently finished implementation of private messaging and the groups dashboards on SUMO, and will meet on IRC, in #testday (please note, it’s not #sumo).

Especially for private messaging it’s important to have many people on IRC at the same time. So if you can manage it, please join us between 3pm and 7pm UTC , but of course we will be around all day. Also, we have a test plan ready, so you can systematically check if things are working as they should.

If you are a localizer, you can also request the group dashboard feature. This will add another tab to your dashboard and those of your team members. That tab will hold your localization dashboard and a message on top that you as the locale leader can change to message your team. Also, your localization group will get a profile page listing the locale leader and every member of the team. You can see here what the German localization group profile looks like.

For the groups dashboard I wrote a short tutorial that you can see here.

Please report any issues that you encounter on this etherpad so we can fix them quickly.

All info about the testday:

Date: Friday, July 8th
Time: 3pm to 12pm UTC
Place: IRC #testday (please note, it’s not #sumo)
Testplan: https://wiki.mozilla.org/QA/Execution/Web_Testing/SUMO/2.8
Feedback: http://etherpad.mozilla.com:9000/kitsune-groups-and-messages

Thanks, and hope to see you later today!

SUMO is adding new features: group dashboards and private messaging

Hello everyone,

Starting today we will be beta testing the new groups and private messaging features of Kitsune. We want to start with a small number of people for now, so we are going to cap this at about ten. We will have a test day for everyone on July 8th. If you are interested in testing private messaging and giving feedback on it, please let me know in this thread on the support community forum and I’ll activate it for you.

If you are a localizer, you can also request the group dashboard feature. This will add another tab to your dashboard and those of your team members. That tab will hold your localization dashboard and a message on top that you as the locale leader can change to message your team. Also, your localization group will get a profile page listing the locale leader and every member of the team.

You can see here what the German localization group profile looks like.

For the groups dashboard I wrote a short tutorial that you can see here.

Private messaging should be fairly intuitive. Once the feature is activated, just click on “Inbox” on the upper right corner of the page. Make sure that you only message people who are listed in the forum thread for now.

Please report any issues that you encounter on this etherpad so we can fix them quickly.

And of course, please let me know if you have any questions.

Thanks!
Kadir

Localization for Firefox 5 and beyond on SUMO

If you are a SUMO localizer now is a good time to prepare for the upcoming Firefox 5 release. Firefox 5 will be released on June 21, which is less than 2 weeks away and our articles are now ready to be localized. There are 7 updated articles for Firefox 5, and many of the updates are rather small, so you should have no problem updating your articles if you had them up to date for Firefox 4 already. As we have discussed before, in the future there will be 4 weeks of localization time beginning from the third week of the Beta period for every new Firefox release. But for Firefox 5 the time for localization is a bit shorter since the whole Firefox 5 development cycle is kept shorter than normal.

Michael Verdi has create a special page that let’s you easily keep track of all the articles that we have changed between Firefox 4 and Firefox 5. If you are a localizer you should bookmark that page so you can see at a glance the upcoming changes for future versions of Firefox.

As always your Localization Dashboard is the most important metric to see what to localize or update next. If you keep the top 20 articles localized at all times you cover over 50% of all visits to the KB already. But with upcoming new Firefox versions we can sometimes predict that an article is going to be very popular after the release, even if it doesn’t show up on the Localization Dashboard yet. So we’ll make sure to inform everyone about those cases.

If you have any questions about the process, an update is unclear or you want to help localize support articles into your own language, please don’t hesitate to contact us in the community forums.

Documentation Process

HOLY COW! This is a long (12 minutes) explanation of the new SUMO documentation process. Here’s the short version:

Six week schedule. The first four weeks, we’ll be drafting articles based on the Aurora release and localizing articles for the Beta release. During the last two weeks Aurora becomes Beta, Beta becomes the new release and we’ll finalize the docs we drafted in the first four weeks. Rinse and repeat. We’ll use the Article Tracking page to organize everything.

Long version

Example schedule: Weeks 1 – 4

Release = Fx 6, Beta = Fx 7, Aurora = Fx 8, Nightly = Fx 9

  • Localize Fx 7: Once the articles are finalized, there will be four weeks to localize before release. We’ll have marked revisions as ready for localization so they’ll show up on Localization dashboards. Also, Localizers can keep abreast of what changes are happening by following the Article Tracking page.
    • Note: In the two weeks previous to this we may have also made updates based on feedback from the Fx 6 release that will also be included in this batch of articles to localize.
  • Draft Fx 8: We’ll have four weeks to draft new articles and updates to existing articles. We’ll be working from the Article Tracking page which we’ll have been updating since the previous documentation cycle.
  • Track Fx 9: This release will be on the Nightly channel. We’ll follow the Release Tracking page to identify which changes and additions will need documentation. Articles that will need to be updated or proposals for new articles should be added to the Article Tracking page along with links to the appropriate feature page and general notes about what will need to be documented. The details of this should be discussed in the Article’s forum thread also linked from this page. The tracking of Firefox 9 features is an ongoing process that will continue through it’s Beta phase but after it moves to Aurora the main changes should only be the possible removal of a feature.
  • Archive: Since Firefox and our documentation will be changing as new versions are released, we’ll use this time to evaluate whether there are any articles that are:
    • Obsolete because Firefox has changed
    • Obsolete because they get so few views as to not be of concern to the vast majority of Firefox users.

Example schedule: Weeks 5 & 6

Release = Fx 7, Beta = Fx 8, Aurora = Fx 9, Nightly = Fx 10

  • Update Fx 7: Firefox 7 will have been released so, if necessary, we’ll be able to go back and make adjustments to our documentation based on feedback from the support forums and live chat.
  • Finalize Fx 8: Once the release moves to the Beta channel, we’ll have two weeks to finalize these articles that were drafted. During this time we’ll:
    • Add screenshots and screencasts if necessary
    • Review and approve the articles
    • Mark the final Fx8 revision as Ready for Localization
  • Begin Tracking Fx 10: This release will now be on the Nightly channel.

More details, including information about the Article Archive and Ready for Localization features are on the Mozilla wiki.

Join the Firefox 4 for Mobile Localization Sprint today

Hey everyone,

This is a quick reminder about our Firefox for mobile KB localization sprint today , Wednesday, March 30th, from 3pm to 9pm UTC. Yesterday the latest version of Firefox for mobile was released and we have finished work on all mobile articles in the KB. They are ready for localization now.

For a first round of localizations we have selected these top 5 articles:

Of course there are more. If you are done already, and you can find more information and links to a Firefox mobile for your desktop on this page: https://wiki.mozilla.org/Support/Firefox4mobile-sprint

There is one issue with mobile: Screeenshots. Unfortunately you do need an Android phone to take localized screenshots of Firefox. So please, if you have an Android device, put your name on the wiki page and join us on IRC #sumo, to help others taking screenshots in their language. The more people with Android phones are around the better for everyone.

All info about the Firefox Mobile KB sprint:

Date: Wednesday, March 30th
Time: 3pm to 9pm UTC
Place: IRC #sumo
Website: https://wiki.mozilla.org/Support/Firefox4mobile-sprint

Thanks, and hope to see you later today!

SUMO in Ljubljana

Mozilla Balkans meetup group photo
The beautiful city of Ljubljana, shortened to Lublana by its natives (I wonder why), the capital of Slovenia, was the host to the second Balkans Meeting this year. I had the chance to represent the SUMO team this time, and ended up locked up in a cell in Ljubljana’s military prison. Read on, if you want to know why.

Last weekend was the second installment of the Balkans meeting, and this time it was much more hands-on oriented. We had the full day on Saturday to cover a whole range of subjects from SUMO, product L10n, QA to Add-on development. For SUMO this year was a year of changes in many aspects, and during my talk I used the chance to explain why we did what we did, and how we thought it would bring us closer to support each and every one of our 400 Million users worldwide.

The second half of the day was reserved for sprints and hands-on action. Since we had just released a brand new KB, this was an excellent opportunity to see the KB used in real live, and learn from the feedback. The participants worked mostly on the top 20 articles in the KB, which serve almost 50% of all visitors to the KB and have a really high benefit-cost ratio.

Particularly interesting was the feedback, most of it about the actual localization experience in the editor. Generally it was geared towards making the editor more user friendly (smaller fonts for more text, resizing of the editor window etc) and the localization experience smoother. In the latter case we are looking into a few ways to do that already, for example by providing a Google translated text as the base for the initial translation of the English text.

After a pretty solid 9 hours of work we headed for dinner in a traditional Slovenian Restaurant, with horse meat as a specialty. I didn’t try that, but my meal was delicious. There we also met our surprise guest for the evening, the new Mozilla CEO Gary Kovacs, who was adopted as a Balkan member for his Hungarian heritage ;)

Working hard during the SUMO sprint

After the hands-on the day before, Sunday was used for the goal setting process, which involved a review of the goals from the last time and their usefulness. The goal setting itself was pretty great, we had an etherpad where we collected the goals, the document was projected to the big screen then and live edited by a dozen people, which made for interesting discussions.

I’m happy to say that the Balkan communities decided to name the localization of the Top 20 SUMO articles into all Balkan languages as one of those goals, and I think the new KB had quite some influence here. There will be follow up calls to talk about the details of each goal, but all in all they look pretty good, and will surely give everyone enough to work on until the next Balkans meeting (I heard rumors about Athens, but who knows ;)

My cell in Ljubljana

So, how did I end up in a cell? Well, that was thanks to Matjaz, our wonderful host from Slovenia. He organize a hostel for the participants, but not just any hostel, it was a former military prison, taken over by university students and turned into a hostel, with the cells left intact and designed by different architects. It’s considered one of the best hostels worldwide, and I can really recommend spending a night behind bars, it’s unusual to say the least ;)

Big William is watching us

I’d like to thank everyone attending this years Balkan meeting, it was a pleasure to meet and work with you all, and special thanks to our host Mathjaz and the organizers, William and Milos, it was a productive and intense meeting, but with lots of opportunities for informal talks and get-togethers. I really enjoyed it, thanks for putting it all together!