In late 2016, Mozilla embarked on a series of product experiments with our strategic partner Cliqz GmbH, a privacy-centric technology company based in Munich, Germany. The full Cliqz experience includes search recommendations, anti-tracking, and anti-phishing protections, and is best seen in their AMO add-on and standalone desktop and mobile browsers.
We chose to begin our Mozilla/Cliqz collaboration in Test Pilot, our Firefox platform to test new features in front of eager tech-savvy users. Our main goal was to understand how Firefox users valued Cliqz recommendations in the address bar. We removed the other feature components so that they wouldn’t unduly influence our feedback from users. We launched the stripped-down Cliqz version 1 address bar in Test Pilot in January 2017.
Cliqz version 1
Normal Firefox telemetry counts the number of searches users perform in Firefox, such as in the address bar or search bar. Cliqz recommendations take users directly to their destination and also answer some questions directly in the address bar. Therefore, a key Test Pilot hypothesis was that users would perform fewer search engine searches.
Our version 1 launch allowed us to start a telemetry baseline to understand usage data, such as search. With Cliqz, we saw approximately a 25% decline in Test Pilot users’ search engine search vs. their prior activity. This result confirmed our hypothesis that users would see direct answers to a large portion of their queries when using Cliqz.
The version 1 launch also gave us crucial initial survey feedback within our first week. The surveys highlighted some major recurring themes:
- Functionality. Many users saw fewer bookmarks and history results than they normally see in Firefox
- Design. Some users did not like the design of the bar or found it confusing
- Trust. The UI seemed foreign, hence users trusted it less or weren’t confident using it.
- Search status. Cliqz does not change users’ default search engine, but because this was unclear many users complained that they’d lost their favorite search engine.
Our user experience leads pushed for a joint work week to focus on the issues solvable through design. We ignored the requests around bookmarks and history for the time being, since those were out of scope for an initial UX overhaul. We held a joint UX work week with the Mozilla and Cliqz teams in Toronto. In just 4 weeks, the team created a dramatic improvement. We launched the Cliqz version 2 in Test Pilot in mid-March 2017.
The blue autosuggest bar returned to the top of the results pane. This bar exists in Firefox and informs the user that hitting “enter” allows them to search. We restyled the “SmartCliqz” information cards, which includes weather, currency conversion, and sports scores. We added icon styling to indicate bookmarks and open tabs. Iconography, fonts and line-spacing more closely resembled the Firefox Awesome Bar standards. In short: we made Cliqz feel more like Firefox.
The results were dramatic. The version 2 improved the Cliqz experience across every expected measure and even an unexpected one.
- User satisfaction increased vs. the version 1
- There were no more design complaints
- Cliqz recommendation click throughs increased
- Survey mentions of users unable to find their default search were entirely eliminated
- External search referrals increased 7%. While there was no numeric goal, this increase confirmed that users better understood how to use their favorite search engine
- Surprisingly, keyboard navigation increased dramatically. This was not something we anticipated. However, it confirmed that keyboard-centric users were more comfortable and expected the version 2 to behave like Firefox
We’ve since made more improvements to the Cliqz experience. We collaborated with the Cliqz UX team on user research, which lead to many contextual and functional changes. We went back to fix many of the things we noticed in Test Pilot but drew out of scope in the UX overhaul. We improved the frequency, speed, and presentation of history results. We added context to many recommendations, such as adding source and timestamp to currency conversion and weather cards.
Cliqz has taken their vast product improvements and moved them into their core AMO add-on and desktop browser. Our time in Test Pilot was incredibly valuable, but our experimentation here has come to an end. We’ll soon be retiring Cliqz in Test Pilot, and interested users will see an opt-in offer to install the full AMO experience.
Thank you to all of our Firefox Test Pilots!