{"id":2400,"date":"2010-03-02T18:37:14","date_gmt":"2010-03-03T01:37:14","guid":{"rendered":"http:\/\/blog.mozilla.org\/metrics\/?p=2400"},"modified":"2019-09-18T12:05:37","modified_gmt":"2019-09-18T19:05:37","slug":"website-optimization-update","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/metrics\/2010\/03\/02\/website-optimization-update\/","title":{"rendered":"Website Optimization Update"},"content":{"rendered":"

While it’s seemingly been quiet on the website optimization front, we\u2019ve been very busy behind the scenes. Over the last few months, Laura Mesa<\/a> has coordinated the design of 5 A\/B tests that we’re now in the process of implementing (3 on the First Run page and 2 on the IE download page).<\/p>\n

Additionally, we’ve expanded the scope of our testing efforts. Look for tests to go live on support.mozilla.com<\/a> and addons.mozilla.org<\/a> this week!<\/p>\n

In the remainder of today’s post, I’ll discuss a few experiment results and share our website optimizations plans. <\/p>\n

Experiment Results<\/b>
\nGetting Started<\/i>
\nIn this test, the Marketing team wanted to determine how adding Firefox tips and tricks<\/a> to the Getting Started page<\/a> would affect user behavior.<\/p>\n

Unfortunately, after running an A\/B test, we didn’t see any improvement in visits per user, pageviews per visit, average time per visits, or bounce rate. For the most part, the differences were minor. The only statistically significant decrease (at the 99% confidence interval) was in visits per user.<\/p>\n

\"\"<\/a>
\n\"\"<\/a><\/p>\n

First Run Design #1<\/i>
\nIn the first of 3 A\/B tests on the First Run page (bottom image), 6.3% more users interacted with the page and total interactions increased 82.4%. The early results are encouraging, but we need to run the test longer to achieve statistical significance. <\/p>\n

Note that the design with more interactions per user isn’t necessarily better. Our final analysis will focus on specific outcomes (i.e. Personas installed and clicks on the “stay connected” buttons). <\/p>\n

\"\"<\/a>
\n\"\"<\/a><\/p>\n

StumbleUpon Promotion<\/i>
\nFor this test, we hypothesized that promoting a specific Add-on would be more effective than promoting Add-ons generally. Currently, both the experimental variation and control promotions have a .7% click through rate.
\n\"\"<\/a><\/p>\n

Surveys<\/i>
\nIn addition to running tests, we used our (highly recommended) website optimization tool<\/a> to run two simple surveys. With the first, we learned<\/a> that 47.8% of users open new empty tabs from the new tab button, 30.4% open from the file menu, and 21.8% open from a keyboard shortcut.
\n\"\"<\/a><\/p>\n

With the second, we learned that 46.9% of First Run<\/a> visitors are first time users, 8.5% used Firefox for under 1 year, and 44.6% used Firefox for over 1 year. The response rate for both surveys exceeded 15%.
\n\"\"<\/a><\/p>\n

Have an idea for a survey you’d like to run on mozilla.com? Let us know in the comments! <\/p>\n

Upcoming Tests<\/b>
\nTwo Additional First Run Designs<\/i>
\nWe will launch 2 additional A\/B tests on the First Run page. Both use tab oriented designs.\"\"<\/a>
\n\"\"<\/a><\/p>\n

IE download page<\/i>
\nWe will test at least 4 new designs for the the IE download page<\/a>. We are still in the design phase, but expect to push the first experiments live next week.
\n\"\"<\/a>
\n\"\"<\/a><\/p>\n

IE Multivariate Test<\/i>
\nIn addition to testing entirely new designs, we want to understand which current design elements are most effective. Accordingly, we will run a multivariate test, switching in and out 4 page elements. Look for a blog post discussing the results soon.
\n\"\"<\/a><\/p>\n

AMO Landing Page<\/i>
\nIn our first addons.mozilla.org<\/a> test, we will measure how the featured Add-ons promotion affects bounce rate and Add-ons installed per visit.
\n\"\"<\/a>
\n\"\"<\/a><\/p>\n

SUMO Landing Page<\/i>
\nThe SUMO team wants to know whether we should reword article titles as questions. Our first SUMO test will do just that. <\/p>\n

In addition to running these tests, we plan to test the download confirmation page and run 3 A\/B tests on the Update page<\/a>. <\/p>\n

Many thanks go out to Laura, John Slater, Steven Garrity, Royal Order<\/a>, and everyone else involved in this process! Next up, I’ll suggest a streamlined process for proposing and running tests. <\/p>\n","protected":false},"excerpt":{"rendered":"

While it’s seemingly been quiet on the website optimization front, we\u2019ve been very busy behind the scenes. Over the last few months, Laura Mesa has coordinated the design of 5 A\/B tests that we’re now in the process of implementing … Continue reading<\/a><\/p>\n","protected":false},"author":1557,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[3668],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/metrics\/wp-json\/wp\/v2\/posts\/2400"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/metrics\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/metrics\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/metrics\/wp-json\/wp\/v2\/users\/1557"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/metrics\/wp-json\/wp\/v2\/comments?post=2400"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/metrics\/wp-json\/wp\/v2\/posts\/2400\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/metrics\/wp-json\/wp\/v2\/media?parent=2400"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/metrics\/wp-json\/wp\/v2\/categories?post=2400"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/metrics\/wp-json\/wp\/v2\/tags?post=2400"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}