Category Archives: User Advocacy

Firefox Sentiment Reports

Hello all from the User Advocacy team! We have a special new tool to discuss today that will help us gauge the impact of each Firefox release on our amazing user base. We call this the User Sentiment Report (USR). While we’ve been working on this internally for the past couple of releases, fine tuning and tweaking the report, we are finally ready to make this report publicly available for both Desktop and Android Firefox 21!

What is a Sentiment Report?

The User Advocacy team spends a great deal of our time reading and tracking the feedback from our hundreds of millions of users via several channels (SUMO, Input, etc.). We use this feedback to find pain points, problems, and pleasures that users have with our products and then report on them to make sure our products get better and better with each release. One challenge that we have is getting the big picture of all feedback for a release, and gauging the general feeling of our users, if a release had a lot of problems for users, or if it was a very smooth one. So, the Sentiment Report was created. This report allows us to see specific releases, the general feeling of users for that release (based on the number of SUMO reports, negative pieces of input, etc.) and easily compare it to previous releases (allowing us to determine what a “Normal” release is). This report is generated at the end of every release cycle using feedback gathered since release day. The insights from this report have not been possible ever before, so we are excited to have expanded this report to Firefox for Android.

Firefox for Android Sentiment Report

While the Firefox for Desktop Sentiment Report has been ongoing for a couple of releases as we fine tuned it, we haven’t had a Firefox for Android report until this cycle. Using what we have learned from Desktop, here is the first ever Mozilla User Sentiment Report for Android!

There are a few things I’d like to highlight about the report. First is the trending topics. On input.mozilla.org we receive thousands of pieces of feedback for each release of Firefox. It is impossible to go through and read all this feedback manually. So with the help of the metrics team, we created an auto-tagger which, using a training set made by hand, will automatically tag all feedback that comes through input into roughly 20 different buckets (Crashes, websites issues, Flash issues, etc.). This allows us to watch for spikes and drops in various types of feedback and chase down anomalies when they happen. In the sentiment report, you can see the current top ten categories compared with the last 5 releases (Blue points are downward trends, Red are upwards). The numbers you see are negative input per 10 million ADI (Active Daily Installs).

Cost of Support (COS). This is an important metric. In it, we track clicks to the Help button in Firefox for Android per thousand ADI’s. The reason we watch this number is if we see a spike in Help button clicks along with a corresponding decrease in ADI’s, it is likely that there is a significant problem in that version of Firefox that is causing us to lose users. You can compare the week by week cost of Support and the overall COS for each version. Lower is obviously better (Low clicks to the help button with high ADI’s).

Most Painful Issues for Top Devices. Another awesome thing the auto-tagger lets us do is track what devices are having more issues in what categories, and lets us track these per release. So for example, we can see that the Nexus 7 was having major issues with Crashing in previous releases, but that this has consistently been trending downwards, meaning the crash work we have been doing has been paying off. Conversely we can see that the Asus TF300T had a small spike in complaints around slowness in Firefox 21, so we should begin to look into that for future releases. We can track these numbers for any device we have data on, but obviously we don’t have room to show them all in the report.

Firefox for Desktop Sentiment Report

Of course we have this report for Desktop too. Released at the same time as the Android Sentiment report, you can view the Desktop User Sentiment Report! This report tracks many of the same things that the Android report does, but with some notable exceptions.

Trending Topics. We don’t have an auto-tagger for Desktop (yet), so this category consists for topics that were automatically generated from feedback we received on this release, and then were manually curated to give the most relevant information possible.

Survey Data. We have a tool on Firefox for Desktop called Startup Snippets. Users that use the default about:home page in Firefox can see these little snippets of text under the search bar. At certain times during a release cycle we will deploy a link to a survey via these snippets to a certain sample of our users, asking for their feedback around the latest version of Firefox. We can then use this survey data to generate a star rating for Firefox (1-5 stars, with 5 being the highest).

The rest of the report is fairly self explanatory, Cost of support being the number of users who click the help button in Firefox per thousand ADI, negative input (and positive input), and the number of new support threads each week. Using all these graphs we can watch how a release fares over every week of the cycle and compare it with previous releases.

The Future

By no means are these reports in their final form. Every cycle of Firefox brings in new improvements to the User Sentiment Reports. We hope to add many new features in the future, such as tracking of Google Play data for Android, an auto-tagger for Desktop, etc. If you have feedback or ideas for these reports please feel free to contact Tyler Downer or Matt Grimes.

Of Course…

None of this would have been possible without a ton of hard work from many different people. Special thanks go to Hamilton Ulmer, Ali Almossawi, Annie Elliott, Ibai Garcia, and countless other people for their feedback and ideas.

User Sentiment Report

We love to measure things here at Mozilla. We measure performance, speed, downloads, installs and more. We look at our successes and our failures. It’s in our DNA to monitor the health of our products to ensure we deliver more Awesome with each release. The User Advocacy team feels that the health and happiness of our users is just as critical to our success as the health of our products. After all, what would we be without our 400 Mil+ amazing users?

We’d like to introduce you to a new project we’re calling the User Sentiment Report. We have been working closely with the Metrics team for several months to create a real time snapshot of our user’s joy and pain. This report will provide a concise overview of the current release and allow for comparisons to previous releases. You will be able to see at a glance where we can focus our efforts to delight our users.

The goal of this project is to provide a User Sentiment Report once per release cycle. We will continue to work with the Metrics team to improve the accuracy and actionablity of the report in 2013. As we gain insights through this work we will also be able to attempt forecasting based on prior releases. That means that we’ll be able to make more informed decisions based on our pre-release feedback and prevent chemspills.

You can find the User Sentiment Report for Firefox 17 here:

http://blog.mozilla.org/sumo/files/2012/12/mozilla_user_sentiment_17.pdf

I’d also like to take the opportunity to thank Hamilton, Ali, Annie, and Gilbert from Metrics as well as Cheng and Tyler from User Advocacy. This project would not have been possible without the dedication and expertise they provided. This has been a great example of how much we accomplish when we work collaboratively!