I’ll start part four of this blog series with an important question: Are our Firefox users happy about their browsing experience? If they are, exactly how happy are they? Can we measure that somehow? If they’re actually not happy, can we figure out why so we can do something about it?
By looking at the right set of metrics, SUMO can really have a finger on the pulse of the large Firefox user base. Analyzing and understanding the data we’re gathering is critical to the continued success of Firefox and will undoubtedly benefit the Mozilla project as a whole.
SUMO is a great source of information here because it’s the obvious channel for mainstream users to look for support and documentation. We are already analyzing the data we’re gathering, but we’re not really there yet. Some information that would help paint a clearer picture:
- Track search terms that have increased in popularity in recent time, rather than just looking at the fairly static list of top search terms as we’re doing today. That could give us early warnings about emerging issues or trends.
- Thorough data mining –- if we dig deeper to understand how our users use the site by tracking data like “Which articles did users read and what did they search for before ending up in the forum/chat?”, we would gain a much better understanding of how well our SUMO website works for our users. This knowledge could give us a clear idea of how to improve Knowledge Base articles and search results.
Today we’re doing a lot of the information gathering manually. To speed this up and allow us to focus on analyzing the data instead, we should automate the information gathering as much as possible:
- Automatically track specific keywords related to “badware” (spyware, adware, trojans, etc.) to ensure we can become proactive against new issues.
- Add alarm triggers when signs of unusual traffic emerges, e.g. excessive reports of tracked keywords.
- Automatically collect metrics to be analyzed daily/weekly.
Measuring SUMO performance
There are many things we can measure to get a better understanding of our strengths and areas of improvement, such as:
- average rating of Knowledge Base article (poll “Was this article easy to understand?”)
- average response waiting time in the Support Forum
- ratio of threads resolved in the Support Forum
- average Live Chat session length and waiting time
- ratio of Live Chat sessions resolved
We are already measuring some of these, but not all of them. Apart from getting all this data in the first place, we should put more focus on presenting it to our contributors, and analyzing it to better understand how we can continuously improve our support and as a result get happier users.
Measuring user happiness
Back to the original question: are our users happy, and how can we measure it? Well, what we can do is measure how well SUMO meets our users’ expectations. If we’re doing well according to our users, that means we have happy users.
To get an overview of our quality of support and whether or not it’s improving over time, we should start measuring customer satisfaction, but since we don’t have customers, let’s just call this user happiness instead! With this data, we will be able to tell how well we meet or surpass our user’s expectations. It can also indicate how well the various components of the site perform.
- The data could show a total average of our user’s happiness level with SUMO as a whole, but it could also show per-component (KB/forum/chat), and even per-article (KB) data, to give us both an overview of, and detailed insights about how well we’re doing.
- The data should be collected automatically and shared quarterly with e.g. marketing team, who are also keen on getting deeper insights about our user satisfaction.
With all of the above, the SUMO community should be able to tell — with confidence — whether or not our users are as happy as we want them to be.