The vision for SUMO – Part 4: Having our finger on the pulse

I’ll start part four of this blog series with an important question: Are our Firefox users happy about their browsing experience? If they are, exactly how happy are they? Can we measure that somehow? If they’re actually not happy, can we figure out why so we can do something about it?

By looking at the right set of metrics, SUMO can really have a finger on the pulse of the large Firefox user base. Analyzing and understanding the data we’re gathering is critical to the continued success of Firefox and will undoubtedly benefit the Mozilla project as a whole.

SUMO is a great source of information here because it’s the obvious channel for mainstream users to look for support and documentation. We are already analyzing the data we’re gathering, but we’re not really there yet. Some information that would help paint a clearer picture:

  • Track search terms that have increased in popularity in recent time, rather than just looking at the fairly static list of top search terms as we’re doing today. That could give us early warnings about emerging issues or trends.
  • Thorough data mining –- if we dig deeper to understand how our users use the site by tracking data like “Which articles did users read and what did they search for before ending up in the forum/chat?”, we would gain a much better understanding of how well our SUMO website works for our users. This knowledge could give us a clear idea of how to improve Knowledge Base articles and search results.

Today we’re doing a lot of the information gathering manually. To speed this up and allow us to focus on analyzing the data instead, we should automate the information gathering as much as possible:

  • Automatically track specific keywords related to “badware” (spyware, adware, trojans, etc.) to ensure we can become proactive against new issues.
  • Add alarm triggers when signs of unusual traffic emerges, e.g. excessive reports of tracked keywords.
  • Automatically collect metrics to be analyzed daily/weekly.

Measuring SUMO performance

There are many things we can measure to get a better understanding of our strengths and areas of improvement, such as:

  • average rating of Knowledge Base article (poll “Was this article easy to understand?”)
  • average response waiting time in the Support Forum
  • ratio of threads resolved in the Support Forum
  • average Live Chat session length and waiting time
  • ratio of Live Chat sessions resolved

We are already measuring some of these, but not all of them. Apart from getting all this data in the first place, we should put more focus on presenting it to our contributors, and analyzing it to better understand how we can continuously improve our support and as a result get happier users.

Measuring user happiness

Back to the original question: are our users happy, and how can we measure it? Well, what we can do is measure how well SUMO meets our users’ expectations. If we’re doing well according to our users, that means we have happy users.

To get an overview of our quality of support and whether or not it’s improving over time, we should start measuring customer satisfaction, but since we don’t have customers, let’s just call this user happiness instead! With this data, we will be able to tell how well we meet or surpass our user’s expectations. It can also indicate how well the various components of the site perform.

  • The data could show a total average of our user’s happiness level with SUMO as a whole, but it could also show per-component (KB/forum/chat), and even per-article (KB) data, to give us both an overview of, and detailed insights about how well we’re doing.
  • The data should be collected automatically and shared quarterly with e.g. marketing team, who are also keen on getting deeper insights about our user satisfaction.

With all of the above, the SUMO community should be able to tell — with confidence — whether or not our users are as happy as we want them to be.

2 thoughts on “The vision for SUMO – Part 4: Having our finger on the pulse

  1. PAStheLoD

    The problem I see with software in general nowadays is tha lack of “fast response”.

    A problem arises, that’s T-0.
    Somehow the “devs” get to know about it, usually 1-2 days.
    Eventually it’ll get fixed, or a workaround will get posted in a forum and it’ll just get up high enough in google, so average users will found it. (1-2 weeks)
    But the poor end user is left in the dark for a long time. (Months if we’re lucky.)

    They won’t search bugzilla for it, they don’t know what went wrong, what to put into the search query, they’re not on IRC, they don’t read Planet Mozilla (even if they do, they certainly aren’t reading all of the posts). They just notice, that “oh, Firefox N+1 does X thing right, that was broken in version N.”

    Or they don’t, because they get frustrated with “technology” and just .. watch TV or whatever.

  2. Andrews

    Another alternative to proactive online chat – offering assistance tool available on the net is mvispy.com I have been using the service for past few months and it has worked out well for me. I have been able to do live user tracking. This is an excerpt from their website.

    If you see a visitor moving rapidly from page to page, a proactive live chat asking “Do You Need Assistance” is an appropriate question to any visitor on your site that seems lost. MVI SPY enables you to do just that from your office computer. You can watch live visitors actions on your site and if they look lost you can send them a ‘can we help?’ message. The message could be canned or unique to each visitor. If you see a visitor pausing on the checkout process you can ask if they would like assistance. Overall, being proactive with your visitors will increase close ratio.

Comments are closed.