What’s up with SUMO – June 24

Big things this week

Firefox Sentiment Reports

Hello all from the User Advocacy team! We have a special new tool to discuss today that will help us gauge the impact of each Firefox release on our amazing user base. We call this the User Sentiment Report (USR). While we’ve been working on this internally for the past couple of releases, fine tuning and tweaking the report, we are finally ready to make this report publicly available for both Desktop and Android Firefox 21!

What is a Sentiment Report?

The User Advocacy team spends a great deal of our time reading and tracking the feedback from our hundreds of millions of users via several channels (SUMO, Input, etc.). We use this feedback to find pain points, problems, and pleasures that users have with our products and then report on them to make sure our products get better and better with each release. One challenge that we have is getting the big picture of all feedback for a release, and gauging the general feeling of our users, if a release had a lot of problems for users, or if it was a very smooth one. So, the Sentiment Report was created. This report allows us to see specific releases, the general feeling of users for that release (based on the number of SUMO reports, negative pieces of input, etc.) and easily compare it to previous releases (allowing us to determine what a “Normal” release is). This report is generated at the end of every release cycle using feedback gathered since release day. The insights from this report have not been possible ever before, so we are excited to have expanded this report to Firefox for Android.

Firefox for Android Sentiment Report

While the Firefox for Desktop Sentiment Report has been ongoing for a couple of releases as we fine tuned it, we haven’t had a Firefox for Android report until this cycle. Using what we have learned from Desktop, here is the first ever Mozilla User Sentiment Report for Android!

There are a few things I’d like to highlight about the report. First is the trending topics. On input.mozilla.org we receive thousands of pieces of feedback for each release of Firefox. It is impossible to go through and read all this feedback manually. So with the help of the metrics team, we created an auto-tagger which, using a training set made by hand, will automatically tag all feedback that comes through input into roughly 20 different buckets (Crashes, websites issues, Flash issues, etc.). This allows us to watch for spikes and drops in various types of feedback and chase down anomalies when they happen. In the sentiment report, you can see the current top ten categories compared with the last 5 releases (Blue points are downward trends, Red are upwards). The numbers you see are negative input per 10 million ADI (Active Daily Installs).

Cost of Support (COS). This is an important metric. In it, we track clicks to the Help button in Firefox for Android per thousand ADI’s. The reason we watch this number is if we see a spike in Help button clicks along with a corresponding decrease in ADI’s, it is likely that there is a significant problem in that version of Firefox that is causing us to lose users. You can compare the week by week cost of Support and the overall COS for each version. Lower is obviously better (Low clicks to the help button with high ADI’s).

Most Painful Issues for Top Devices. Another awesome thing the auto-tagger lets us do is track what devices are having more issues in what categories, and lets us track these per release. So for example, we can see that the Nexus 7 was having major issues with Crashing in previous releases, but that this has consistently been trending downwards, meaning the crash work we have been doing has been paying off. Conversely we can see that the Asus TF300T had a small spike in complaints around slowness in Firefox 21, so we should begin to look into that for future releases. We can track these numbers for any device we have data on, but obviously we don’t have room to show them all in the report.

Firefox for Desktop Sentiment Report

Of course we have this report for Desktop too. Released at the same time as the Android Sentiment report, you can view the Desktop User Sentiment Report! This report tracks many of the same things that the Android report does, but with some notable exceptions.

Trending Topics. We don’t have an auto-tagger for Desktop (yet), so this category consists for topics that were automatically generated from feedback we received on this release, and then were manually curated to give the most relevant information possible.

Survey Data. We have a tool on Firefox for Desktop called Startup Snippets. Users that use the default about:home page in Firefox can see these little snippets of text under the search bar. At certain times during a release cycle we will deploy a link to a survey via these snippets to a certain sample of our users, asking for their feedback around the latest version of Firefox. We can then use this survey data to generate a star rating for Firefox (1-5 stars, with 5 being the highest).

The rest of the report is fairly self explanatory, Cost of support being the number of users who click the help button in Firefox per thousand ADI, negative input (and positive input), and the number of new support threads each week. Using all these graphs we can watch how a release fares over every week of the cycle and compare it with previous releases.

The Future

By no means are these reports in their final form. Every cycle of Firefox brings in new improvements to the User Sentiment Reports. We hope to add many new features in the future, such as tracking of Google Play data for Android, an auto-tagger for Desktop, etc. If you have feedback or ideas for these reports please feel free to contact Tyler Downer or Matt Grimes.

Of Course…

None of this would have been possible without a ton of hard work from many different people. Special thanks go to Hamilton Ulmer, Ali Almossawi, Annie Elliott, Ibai Garcia, and countless other people for their feedback and ideas.

What’s up with SUMO – June 17

Big things this week

What’s up with SUMO – June 10

Big things this week

  • Notes and video from this SUMO meeting.
  • Firefox for Android user sentiment report coming soon. We will blog about it when it’s done.
  • Firefox OS translations are going great! Thanks everyone :)
  • Contributors of the week: Uptown22 (7 solutions in one day), colbabomb (made top 25 in first week).
  • Current SUMO development sprint – 2013.12
  • Next SUMO meeting – Monday, Jun. 17th (call in details & meeting notes) at 9:00 am PDT. Please add your comments, questions and updates to the wiki. You can also participate in #sumo during the meeting. We’re going to record and post a video of the meeting.

SUMO Day and SUMO bug triage day happening today!

Today is a special day! As we’re continuing our tradition of having a SUMO Day every other Thursday, we are also joining efforts with the bugmasters team to clean out SUMO bugs.

We’ll be answering questions in the support forum AND we will be helping Liz triage around 600 SUMO bugs that are currently open. You can find more details about this on the bug days wiki

Join us, create an account and then take some time on Thursday to help with unanswered questions. Please check the etherpad for additional tips.  Our goal is to respond to all the questions posted on Thursday, so please try to answer as many questions as you can throughout the day. Also please join #bugmasters channel on IRC to connect with the bugmasters team and help out with the triage.

Don’t forget: if you get stuck or a question is too tricky, we’re there to help on #sumo

Happy SUMO day and SUMO bug triage day everybody!

Sumo bug day

 

What’s up with SUMO – June 3

Big things this week

  • Notes and video from this SUMO meeting.
  • In addition to SUMO Day this Thursday, we’re going to be triaging SUMO bugs. Join us!
  • Contributors of the week: Amit, Jafar Muhammed (Jafar Muhammed) and Jayakumar Sadhasivam (iamjayakumars). Congratulations!
  • Star Trek thread.
  • Current SUMO development sprint – 2013.11
  • Next SUMO meeting – Monday, Jun. 10th (call in details & meeting notes) at 9:00 am PDT. Please add your comments, questions and updates to the wiki. You can also participate in #sumo during the meeting. We’re going to record and post a video of the meeting.

It’s Thursday, it’s SUMO Day!

A new SUMO day is here!

Every other Thursday we’re organizing a SUMO day where we try to answer all the questions coming on the support forum. This Thursday, May 23rd is a new SUMO day!
We’ll be answering questions in the support forum and helping each other in #sumo on IRC from 9am to 5pm PST (UTC -8).

Join us, create an account and then take some time today to help with unanswered questions. Additional tips for getting started are on the etherpad. Our goal is to respond to every new question posted Thursday, so please try to answer as many questions as you can throughout the day.

Last time, we managed to reach 91% of questions answered! Can we reach 100% ?

Do not forget:

Happy SUMO Day!

 

What’s up with SUMO – May 13

Big things this week

What’s up with SUMO – May 6

Big things this week

  • Notes and video from this SUMO meeting.
  • Work continues on new Topic and Subtopic information architecture. Hopefully there will be designs ready to share later this week.
  • Tom’s presentation about live chat (starts at 4:47).
  • Brazilian Portuguese community reboot is amazing. If you want to reboot your community or get it to be more active, contact us and we’ll be more than happy to help.
  • Current SUMO development sprint – 2013.9
  • Next SUMO meeting – Monday, May 13th (call in details & meeting notes) at 9:00 am PDT. Please add your comments, questions and updates to the wiki. You can also participate in #sumo during the meeting. We’re going to record and post a video of the meeting.

What’s up with SUMO – Apr. 29

Planet Mozilla viewers – you can watch this video on YouTube.

Big things this week

  • Notes and video from this SUMO meeting.
  • New charts on the KPI dashboard. Let Kadir know if you run into issues with them.
  • Bram is working on a draft of the new product and topic landing pages to support the topic and subtopic hierarchy. Michael will be placing articles in that new hierarchy this week.
  • Documentation statusboard – Keep up with all of the current documentation projects.
  • Next SUMO development sprint – 2013.9
  • Next SUMO meeting – Monday, May 6th (call in details & meeting notes) at 9:00 am PDT. Please add your comments, questions and updates to the wiki. You can also participate in #sumo during the meeting. We’re going to record and post a video of the meeting.