Input: 2014 retrospective

2014 was a big year for Input. This is the year we hit a point where the project is mature and stable. This is the year we worked on making it easier for new people to start working on Input. This is the year we fixed all the issues that made it difficult for Input to support multiple products. This is the year we created an API that allowed anyone to access the data to let them write their own dashboards. This is the year we implemented a translation infrastructure so that non-English feedback can flow through our tools (previously, it was handled manually). It was a big year!

Read the rest of the retrospective on Will KG’s blog!

Helping our Users Experience the Modern Web

It is a well known fact that users who are running out of date versions of software are at risk to a variety of security attacks. Firefox is no exception. Not only are out of date users at risk to a whole host of known security issues, they are not experiencing the best the web has to offer in terms of modern web technologies, web developer tools, stability and performance.

Unfortunately, not all users are automatically updating. The User Advocacy team noticed that each time we release a new version of Firefox, around 2% of our users get stuck on that version and never update.Firefox Version Adoption

(Photo Sourced from ArsTechnica and NetMarketshare)

While Firefox does allow user choice in deciding if they want to keep up to date automatically, install updates manually, or disable them entirely, the vast majority of our users leave updates enabled. In a survey ran last year of users on out of date versions of Firefox, just under 70 percent of users surveyed not only wanted to be kept up to date automatically, but thought they were already on the latest version. Obviously, something was going on and these users weren’t updating automatically like they should have been.

The Solution

After running this survey in Q2 of 2013, the User Advocacy team sat down with various stakeholders throughout the organization such as Robert Strong, who owns the Firefox Updater mechanism, Benjamin Smedberg, various members of the Metrics team, and others. Between us, we formed the idea to give out of date users a hotfix that bypasses the built in updater and downloads a fresh installer of the most recent version of Firefox, assuming that they’re on a new enough version to support Firefox’s Hotfix system (Firefox 10 and up). Thus was born Bug 928173. This hotfix was completed and released to users on July 16.

Some items of note about the Update Hotfix:

  • It is currently Windows only, and will only operate on Non-ESR versions of Firefox 10 and above (We will continue to expand the audience that can receive this hotfix in the future)
  • It will only attempt to run on a Firefox profile that has updates enabled (we want to respect user sovereignty)
  • If a user fails to allow the UAC prompt the hotfix will display a prompt to the user until they install the update. We expect to see the downward trend of out of date users continue for some time as users allow the update to happen.
  • We’ve found a few bugs in this hotfix, and will be investigating a v2 to fix them.
  • Due to a bug in our data collection, we lost the first 24 hours of hotfix deployment numbers. This is unfortunate as we suspect the majority of activity happened in the initial 24 hours. We will try to estimate these numbers with other data.

The Results

Hotfix Deployment Results:

As of July 28, well over 8.4 million users have installed the Add-on Hotfix (Note, this is missing the first 24 hours of data, so the number is probably significantly over 8 million). Of the users who have installed the add-on, over 4.3 million have actually updated (again, likely more). Of the remaining users, the majority didn’t allow Firefox to update when the User Account Control (UAC) prompt appeared, and so they will receive a notification asking to update Firefox until they end up installing the update. There are a few bugs we’ve found in the hotfix (more on that later), but we believe the vast majority of installs are proceeding smoothly.

SUMO:

For users experiencing trouble with the hotfix, we direct them to the SUMO website. We have a special article written just for users who are having a hard time updating. Prior to the hotfix, this article was receiving a few hundred users a day (it receives more on a week where we release a new version of Firefox, but that was several weeks before this hotfix went live). The graph below represents page views after releasing the hotfix:

Screen Shot 2014-07-25 at 1.22.47 PM

In just 24 hours, we went from around 400 visitors a day to nearly 600,000 users in one day!

Missing Users and Total Impact:

Using our numbers of all users who installed this add-on, and visited the SUMO site, we can estimate that somewhere in the area of 11.5 million users have installed this hotfix so far, with around 6 million updates installed! Since were were so restrictive in how we deployed this hotfix, we will be able to use these numbers going forward to better estimate our out of date population and how to help those users.

Moving Forward

We are extremely happy with the results of the Hotfix so far, but there is still work to be done. Work on V2 of the hotfix, is already underway and will fix some of the bugs we’ve discovered in the current hotfix:

Please stay tuned for future updates. If you do have other questions feel free to reach out to the User Advocacy team or any of the stakeholders above.

User Advocacy Q2 Roundup

Q2 2014 was a very busy and productive quarter for the User Advocacy team. While we completed too many things to count, I’m going to try to call out some of the bigger projects we made significant progress on in Q2, and some of the stuff that we will be carrying over to Q3.

Australis:
Being the biggest redesign since Firefox 4, Firefox 29 naturally took up a significant amount of our (and everyone’s) time this quarter. For the User Advocacy side, we spent a significant amount of time researching our pre-release audience. We ran multiple surveys in Nightly, Aurora and Beta, studied how well users liked the update, the First run tour, etc. All of these pre-release surveys lead up to the big release, which lead to a fury of reading input data, the SUMO forums, and helping gather feedback for the eventual 29.0.1 release we ended up releasing. Overall, this long process, and cross-team work that we did, has helped us not only understand our users better and serve them a better product, but has helped us learn and grow for future large projects like this in the future.

You can read more about our feedback strategy for Firefox 29 here.

Firefox Updates:
Another large user facing project we have been pushing for over the last quarter has been the Firefox Updates Hotfix. This is a large hotfix that will be deployed to over 20million+ users who are not on the current version of Firefox, and will help them to update to the latest version of Firefox easily. While the hotfix hasn’t gone out yet, it will go deploying within the next few weeks. With millions of previously out of date users on the latest version of Firefox, able to use the newest features (Firefox Accounts, Health Report, etc.), on top of enjoying better performance and security. We are very excited for this fix and can’t wait to share the results with you.

If you’d like to track the progress of this Hotfix, please see the bug. Also, watch this blog for future updates.

Firefox Accounts Survey:
Speaking of new features, we always try to find out what our users think about our new features and products. We have recently launched a survey of our Firefox Accounts users. This is a new survey, so we haven’t gotten a complete data set yet, but stay tuned!

Input:
As one of our biggest sources of feedback, Input has a bunch of new features we have been wanting to implement for sometime. Thanks to the awesome Will Kahn-Green, we are making amazing progress on the Input site. You can read Wills great Post-Mortem at http://bluesock.org/~willkg/blog/mozilla/input_2014q2, but pay special attention to the Automated translation piece and the brand new dashboards. It’s exciting stuff.

This list doesn’t even begin to touch the number of things that we have accomplished this Quarter, with dozens of other projects, large and small, being completed in just 3 short months. Overall we are very happy with our quarter and are super pumped for an even more exciting Q3. Stay tuned for more updates on what is coming down the pipeline soon!

Input Status Update – 05/13/2014

It’s been a few weeks since our last post. We’ve been busy!

In the last month or so we went to San Francisco, saw the new 49er stadium, knocked out more team goals, played some board games, expanded the team, went to Portland, did a Pedicab tour, drank lots of good beer, ate lots of good food, played more board games, locked in more Input Roadmap items, and created a ton more work for Will. Wheeeeeew. More information on all that craziness later.

It was an exhausting few weeks, but it didn’t even phase Will “The Juggernaut” KG! Here’s what Will has been up to lately.

Continue reading …

Input Status Update 03/28/2014

In Input development over the last couple of weeks, Will has been making some serious progress. Here’s the latest and greatest:

* [bug 983980] Update favicon.png (Thank you Swarnava!)
* [bug 982242] Add description_terms field
* [bug 949188] Document new strings process
* [bug 979486] Restyle the Firefox OS feedback form (Thank you Rehan!)
* [bug 986071] Fixes the cyoa page
* [bug 980347] Fix domain extraction
* [bug 987172] Fix unicode domain handling

The description_terms field in the index will make for better trending
lists, bigrams, wordclouds, etc. We still need to implement some stop
words related work, but it can be used now.

Rehan finished the style overhaul for the Firefox OS form. The generic
form still needs to be restyled. I’m not sure when this will happen.

Swarnava updated the favicon to the current Firefox logo. Thanks for all the help!

We show the domain of the urls submitted with feedback now. This fixes
some issues where the feedback response was meaningless without the
context of the url it was talking about.

What’s next for Input? We’ve got some big surprises in store. Just keep an eye on this space for more updates!

The next big thing for Firefox and how we listen to our Users

Hello there everyone!

As you may know, today is the release of Firefox 28 (if you haven’t already, go download it and check out the awesomeness!), which means that Firefox Beta will be updated to version 29.  This means that beta users will be upgraded to the redesign of Firefox (code-named Australis), giving us a more widely used platform for testing and feedback. If you aren’t already, please download Firefox Beta and use it, give feedback, and enjoy!

 

User Advocacy for Australis

One of the cool things about Firefox is that we release new versions in stages, called “channels”, so that we can better integrate feedback from our users before launching the final version of the product!  We have four channels of deployment that are released in the following order over 6 week intervals: Nightly, Aurora, Beta, Release.

More blog worthy than even the release of Firefox 28, are the results of our recently completed user research from Firefox Aurora 29.  One new feature in Aurora and inspired by our Nightly feedback was a fantastic UI Tour showing the new features and changes in Australis. We were very interested to investigate how this tour was received by users and how useful it actually is.

 

We surveyed thousands of Aurora users over the past six weeks and in the process found out some very interesting things:

Overall, most users are very happy with the new design both with and without the tour. However, the data shows that the tour does a great job helping with the learning curve  of Australis,  as most of the users who took it said it answered all of the the  questions they had with the new design. The Onboarding team has also made some minor changes to the tour to reflect the feedback that we received, and will be interesting to see how this feedback improves in Beta.

 

While we set out to look at the tour in particular, many of the comments we got were about Australis as a whole, here are a few of the representative comments:

  • Welcome change. The new look is great and ahead of any other browser. Nice work.
  • This might be enough to get me to switch back to Firefox after using Chrome as my default browser for years. Excellent first impressions so far!
  • The new UI is awesome, it’s faster and it doesn’t get in the way of what you want. It’s a fresh new and lean look on what a modern browser should be.
  • Firefox looks ****in awesome now. Really.
  • It’s better faster has more abilities and AMONG ALL MORE FUTURISTIC
  • When I saw this version of Firefox I immediately switched back from Chrome to Firefox. Looks awesome!
  • Slimline menu bars are very well received.
  • I know this interface has taken a lot of flak for being different — but in all seriousness, I have switched from using Chrome to using this.

Most users seem to feel that Australis is a very welcome redesign, with common words being sleek, clean and fast. As these are some of the goals of Australis, it is encouraging to hear that the majority of users  found that Australis meets these goals.

 

Of course, as with any interface change, not everyone is happy. With a change this significant, there are some users who didn’t like Australis for a variety of reasons. Here’s some of the negative feedback that we received:

  • Miss a lot of options, ridiculous big buttons/icons.
  • Allow us to at least move the menu to the left side. Or get it where it belongs.
  • I want to have add-on bar at the bottom! I want the old design back!!!
  • I cant put the tabs via about:config back there they belong – BELOW the bookmarks
  • Seriously, the icons are too large, where is the small icons function? I can’t even move the refresh/stop outside the address bar.
  • I really dislike the removal of the addon bar. It was handy having forecastfox sitting right near the clock for a quick glance at the time and weather.
  • Removing a bunch of possible customization (e.g. Add-ons Bar, the back/forward/stop/reload buttons) and ignoring the OS theme (e.g. arrow panels and removal of Small Icons mode) is bad.
  • I would reintroduce the new menu using the Orange Firefox button to maintain the iconic Firefox brand image.
  • I don’t mind a redesign, but I do mind removing features like addon bar.

As seen in these representative comments, some of the biggest things users miss are the Add-ons Bar, the Tabs on Bottom add-on, and small icon mode. There are some add-ons that bring back some of these features for those users who want them. We are also making some in-product tweaks to help make these users happier.

 

The Future

Going into Beta we aren’t going to slow down. We already have a full battery of surveys ready for Beta, and we will use them to make sure that the changes we’ve made in Aurora continue to make the user’s experience better (Just like the Aurora audience was happier than the Nightly audience). As said before, please download Beta, use it, give feedback, and help us make Firefox better for hundreds of millions of users!

The Transition to a Long-Term Tool Architecture

Historically the User Advocacy team has been relied upon to get user feedback analysis done quickly and effectively.  With a growing reputation, the team has been in high demand for exciting new projects and ad hoc analyses.  However, our velocity on these projects has resulted in growing technical debt (an acceptable trade-off at the time).

Fortunately, with increased headcount (Rob and Aimee), we’re able to re-focus on sustainable solutions to our stakeholders’ problems.  We have the following long-term goals in mind for our projects going forward:

  • Develop a set of standards for how our code is to be organized/written:
  • Become the owners of our tool infrastructure to achieve faster iterations and have more independence:
    • New front-end VM
      • For displaying dashboards/reports to the rest of Mozilla
    • New back-end server
      • For pipelines and fun machine learning projects

If you have any questions on how/why we are making these changes, have any ideas on how we can do things better, or would like to become a technical contributor to these changes, please let us know!

Input Status Update 03/05/2014

It’s that time again! Time for another recap of the new Input awesomeness since the last update. Over the last couple of weeks, here’s what Will_KG has landed in Input:

[bug 843732] Handle utm_source and utm_campaign
[bug 857734] Write better csrf error page
[bug 969667] Clean up CSS/JS
[bug 969667] Replace mobile/desktop forms with generic
[bug 969667] Merge desktop and mobile forms
[bug 970325] Add product picker page prototype

Handling and storing utm_source and utm_campaign querystring parameters
should make it possible to distinguish between organic and non-organic
feedback.

Joshua wrote up a better CSRF error page. Yay Joshua!

Will also did a bunch of code cleanup and merged the desktop and mobile
feedback forms. This was a big rewrite of this code. This allows us to
add new cards in the future, measure drop-off rates and generally makes
maintenance a lot easier for me, testers and translators. Further, I
contend it fixes some of our privacy-notification problems and it adds a
url field to the mobile feedback form which we didn’t previously have.
It also sets us up well for a product picker page which will further
generalize Input and make it more useful.

If you’ve got any other ideas for what you’d like to see in Input, let us know! We are always interested in hearing from the community!

Input Status Update 02/20/2014

Will Kahn-Greene has been hard at work the last couple of weeks landing more Input awesomeness. Here’s a subset of the recent changes:

  • cron support (yay!)
  • l10n_completion tracking
  • [bug 965794] add product table
  • [bug 965798] rework code to use product table
  • [bug 965796] add product list view to analyzer area
  • [bug 933786] fix strip() error on stage
  • [bug 970514] add analyzer search view

I just spoke to Will earlier today and he already has several new items to add to this list. We’ll save those for the next update!