Improving a User’s Experience with Firefox Support (part II)

Ken Kovash

5 responses

We recently discussed an optimization test at the Firefox Support site (support.mozilla.com).  At the time, we mentioned that:

Next up, the SUMO team is planning to soon roll-out a much more substantial A/B test at support.mozilla.com.  With a somewhat radically different page design, our expectations for this next test are different.  In other words, the more variation that is tested generally leads to bigger differences in the results.  Stay tuned!

We just ran this “test #2” over the past week or so and we now have the results to share.  For reference, the full project plan is here and the two page versions we tested are below.

Version #1 (Existing SUMO home page):

ab_test2_page3_blog

Version #2 (New design for SUMO home page):

ab_test2_page4_blog

What did the results say?  Out of nearly 1.5 million visitors, we saw the following behavior:

test_2_final_results

What’s the impact of the “new design” page?  Decreasing the bounce rate (or exit rate) by 0.5% translates to an improved user experience for approximately 260,000 Firefox users (annually).  These are users who would have otherwise immediately exited the SUMO site (some perhaps feeling unsatisfied or unhappy).  The one other critical metric for the SUMO team is the rate at which users utilize the knowledge base search box.  Again, you’ll see that by this measure, the “new design” wins.  11.9% of visitors searched vs. 10.6% on our control (or existing) page design.

Thus, the SUMO team will now be implementing the “new design” as the permanent new Firefox Support home page.

Moreover, we dug into the data here a little further.  We asked ourselves, “of people clicking through or taking some action on either version of the SUMO home page, did one version affect the chances that the user clicked on something beyond their initial action?”  The answer appears to be “yes”.

First, let’s identify what are the other common actions taken by users at our two page versions, and then we can talk through what happened to users after this initial action.

ab_test2_clickthru_actions

The two most frequently clicked links on the existing SUMO home page – “Browsing basics” and “Customizing Firefox” – seem to provide content on their respective pages that is rarely used.  During our test, we saw that less than 10% of users who navigated to “Browsing basics” further clicked on its relevant content.  “Customizing Firefox” performed even worse, as less than 5% of users navigated to any of its content.  Conversely, the two most popular links on our “new design” page – “Browse them all” (Popular Support Articles) and “Cannot clear location bar history” – appear to be clear improvements compared to the existing page.  “Cannot clear location bar history” experienced 25% of its users following a suggested link and “Browse them all” saw 40% of its visitors use one of the suggested “Top Articles”.

On many dimensions, this optimization effort at support.mozilla.com looks like a huge win for Firefox users.  Please click over to the SUMO blog for more details regarding future plans around further user experience improvements.

5 responses

  1. Pino wrote on :

    I think these figures really interesting and think it is a great thing you people test it in such a way what method works best.

    The way I interpret the first table however is that the new page worked better overall, because the search box performed a lot better. The rest of the articles became less interesting on the other hand. This could be due to people normally doing “other” now doing a search, but it could also be new people searching and people who would click on the other pages on the old version, are no longer interested in the new version. (if you understand what I mean, lol) Have you experimented combining both pages, so that you keep the higher ‘other action’ from the old page and get the new search box?

  2. JH wrote on :

    Because an A/B test was run instead of an MVT, you won’t understand WHY recipe B won. Maybe it was the cloud, or the “new to firefox” section. I’m surprised Mozilla isn’t doing something more sophisticated than a simple A/B.

  3. Majken “Lucy” Connor wrote on :

    Can you tell how many people made use of the search suggestions?

  4. Blake wrote on :

    Nice job with the experiment Ken! I’ve created a rough mockup of another SUMO homepage that we may want to test. I think Google does a pretty good job with their support pages and have borrowed a few elements from them, including the prominently featured “Top Articles” section and the link organization. It may also be helpful to use an auto-suggest drop down in the search box.

    Mockup: http://farm4.static.flickr.com/3572/3663094710_57b89c1833_o.png

  5. Donnie Berkholz wrote on :

    You might want to try some real statistics to prove whether the two distributions are actually different. Try Student’s t-test.