A few months ago, David Tenser from our Support team came to the Metrics team with some ideas for analysis concerning user experience with our support.mozilla.com site (SUMO). At the time, we did not have the functionality set up to get the data we wanted, but thanks to some hard work from Jeremy Orem and multiple members of our WebDev team, all of this was made possible about three weeks ago! Now we can see a user’s search term, which article is clicked on after searching, and whether or not the user thought the article was helpful:
The first bit of data we wanted to extract was our Bounce and Refine Search rates for our top search terms in SUMO. For this, I defined a bounce as a SUMO visitor searching a particular term and then immediately leaving the site. Bounces are viewed as failures to give the user results they were looking for. A refined search is defined as a user that searches one term and without clicking an article link or navigating anywhere else searches for a refined version of the same term. Again, we look at a refined search as evidence of a user not receiving the appropriate information with their original search. Here’s what these numbers look like for our top search keywords:
Highlighted in red are the particularly problematic search terms for SUMO in terms of bounce and refine search rates. Private browsing is the most alarming — not only is it our third most popular search term, but it also is in the highest bounce rate and refine search rate groups. From a user standpoint, searching private browsing is most likely an attempt to find out how to turn on private browsing. We can see this in the data; the top refined search for this phrase is start private browsing. Looking at the search results a visitor sees, the only one relevant to private browsing is the top result:
This article does explain how to start private browsing, but the description gives off more of a “what is private browsing” vibe. Either a “Start Private Browsing” article should be started or the phrase should be included in this article’s description.
The bounce rate of home page is a little more perplexing. It seems the main motivation behind this search is to change the default home page in Firefox. Fortunately, the top search result is:
So no issues right? Looking at the refined searches reveals a surprising anomaly: the top two refined searches are “set home page” and “change home page”. I’m baffled. Could users just be skipping past this first result without realizing it? Not likely, but possible. We need to explore other reasons for the high bounce rate — maybe users are searching home page for other reasons.
Next up is history. Users searching this term are (on assumption) likely to be looking to clear their history. Unfortunately, for Firefox 3.0 users there is not a clear result to navigate to. “Clearing Location bar History”, “How to clear Search bar History”, and “Clearing Private Data” are the appropriate articles to continue with, but for inexperienced users who might not know what these terms mean, there is not a definitive how-to-clear-your-history result.
For the mass migration from the downloads search results page (44% bounce rate) I think an article on where to find downloaded files would work well. Right now, there is no result pointing users in this direction, which is probably contributing to such a high bounce rate.
Import bookmarks is up last. Looking at the refined searches, import bookmarks from internet explorer tops the list. Comparing this with the search results, this article is not listed on the front page. I believe moving it up in the line of results could quell the refine search rate.
Each of these changes represents a small change to our database of support articles that could taken together greatly increase the effectiveness of search in the Knowledge Base. In part II of the SUMO analysis, I’ll be taking a look at pathing after searches — the articles visitors are choosing to view after a specific search and their satisfaction with each one.