Firefox Telemetry

Benchmarks Suck

Mozilla has traditionally relied on [Talos, Sunspider, Kraken, etc] benchmarks to optimize Firefox. Unfortunately there are two problems with benchmarks: a) it is hard to write good benchmarks (see all of the complaints about Sunspider) b) the most perfect synthetic benchmarks do not completely correspond to actual user usage. Firefox with a well-used profile, anti-viral software, well-aged Windows and 30 addons will not perform the same as it does in our clean benchmarking environment.

For my team this became obvious in Firefox 4 once we started recording Firefox startup times. Turned out that it is easier to work on fixing startup performance than make our synthetic test closely reflect real world startup speed.

Telemetry

There is only one solution to this problem: develop telemetry infrastructure to measure Firefox performance in the wild. Beginning with version 6, Firefox will ask users to opt-in to sending anonymous usage statistics about performance, user interface feature usage, memory usage, and responsiveness to Mozilla. This information will help us improve future versions of Firefox to better fit actual usage patterns.

This functionality is already present in our major competitors. Unlike our competition we do not plan to tag reported data with unique identifiers. This will make it harder for us see how Firefox performance changes over time for particular users (or easily tell whether some users are disproportionately represented due to sending more reports). We take our users’ privacy seriously, so this seems like a reasonable trade off.

Yesterday I landed the reporting part of telemetry (bug 585196). We are still working on UI, official server-side and on updating the privacy policy.

Above screenshot shows some of the data that will be gathered for users that opt-in to telemetry.

Please help us get a headstart on telemetry. In the recent nightlies, go to about:config and set toolkit.telemetry.enabled to true. Once the pref is set, Firefox will send interesting performance data to the telemetry test server. The metrics are very compact and are sent out no more than once a day.

Since there is no UI yet, install my about:telemetry extension and navigate to “about:telemetry” to see the metrics collected.

30 comments

  1. Awesome job Taras! I hope to try it out in the next few weeks. I’d love to hear what you learned while writing it: do you think your implementation is very different, other than as described above, from Chrome UMA uploads?

  2. Gavin,
    We are in catching up šŸ™‚ Using UMA as a starting point. No point in reinventing the wheel here. The stuff I landed is pretty barebone. I expect us to be on par by Firefox 7.

  3. I’d prefer if the transmission to the server wasn’t over unencrytped HTTP.

    Also, the histograms could do with a bit of help. Labels at the top of each column to explain what they are? Presumably right is frequency and left is some kind of bucketed value with varying bucket-widths? Could the standard deviation be added too, next to the average?

  4. Dan,
    The production server is going to use https.

    I could use some help with presentation šŸ™‚ Patches are welcome.

    Left is bucket label. Buckets are have a scale (linear or exponential). See https://developer.mozilla.org/en/XPCOM_Interface_Reference/nsITelemetry for developer-side of things. I will make a separate blog post on that.

    std dev may or may not be added. I’m basing the first version on chrome code and they took out standard deviation because it wasn’t useful in aggregating large datasets. See version control log for their histogram.cc

  5. It would be nice, if submission of the reports could be limited by connection type. I don’t know if it is even possible for you to get this kind of information from the OS, but I wouldn’t want this data to be sent while I’m connected via UMTS. I guess it will be possible to choose “Always ask”, but as the data will be sent every day, it might be a bit of a hassle to approve this every time. Also true for other things that transmit random data like Test Pilot. I choose always ask for Test Pilot and that was okay, because it didn’t ask me to often. But in this case, I would probably have to opt-out, because the questions would get on my nerves after a few days.

  6. While I think it’s really important to give people a way to help us without being identifiable, I am sure that there are a ton of people out there, who would love to give us the most exact data so we can fix issues like slow start up, excessive memory usage etc. Can’t we have that as an extra checkbox to check?

    [x] Provide data without unique identifiers (more anonymous, less usable data for us)

    [ ] Provide data with unique identifiers (less anonymous, more usable data for us to fix issues)

    [ ] No, thanks.

  7. David Illsley

    I’m interested to know where in the profile the information is stored, or is it information which is already stored by various Firefox components, and is extracted when it decides to submit to the server?

  8. I’d like to second the idea that users be given the option to include an anonymized-but-unique ID to allow for better data collection. Even if only 10% of people opt-in to the higher level of info, that should still give you a good base for watching trends that affect individual installs over time.

  9. That’s interesting. It seems it will make much easier to investigate the dll preload and many others. šŸ™‚

    And a +1 to Kadir’s question.

  10. Oops, i find a little bug in the extension: the about:config link points to about:memory when asking to set the pref on.

  11. Mozilla is probably the one organization with which I’d share usage data.

    It would be a good idea to include an option to allow users to opt-in to sending a unique identifier. You have a good reason to do it, and Mozilla is probably the one organization I’d even consider sending that level of information.

  12. Great job Taras! Do you think it would be hard to add some code so that extensions can hook into this reporting mechanism? I was considering doing something like this for Adblock Plus for a long while, to get some idea about how people really use Adblock Plus. It would make sense to piggyback on the existing mechanism instead of reinventing the wheel.

  13. Then again, the way I read the interface this is already an open API that can be used by any code to add any kind of data. So I should be able to add some Adblock Plus data. Great!

  14. Iā€™d like to be given the option to include an anonymized-but-unique ID to allow for better data collection, too. There is probably no good reason to exclude volunteers wanting to provide more valuable data to the Mozilla developers.

  15. I agree with Kadir. Give us the option to include a unique ID although it shouldn’t be selected by default.

  16. So how does this differ from the existing Test Pilot system? Just that it won’t be an extension?

  17. This is an incredibly important step for improving the user experience of Firefox, awesome work šŸ™‚

  18. Thanks for the positive feedback! To provide a user id, we could provide a freeform text-field for the user to indicate how they can be contacted. This might be useful to identify unique users and to be able to contact users directly to diagnose problems. We sort of do this with the crash dialog.

    Wladimir, good question. There is no technical reason to deny addon authors access to this data. However we’d have to clear this with the privacy people, this might need modifications to the privacy policy, etc. Personally, I would be very happy to help addon authors gauge their impact better.

    Alternatively, if you look at the testcase, you can send a copy of the data to your own server.
    It’s really up to users of whether they trust addon authors with this info. Ie once you install an addon, it can do whatever it likes.

  19. No trouble with unique ID’s, as long as they’re anonymous. They are in all cases I’m aware of. *shrug*

  20. How about some Seamonkey support

  21. Hey Taras, this is really cool stuff! I wonder if we can somehow combine this with the Firefox Endurance Tests project, which uses Mozmill to gather metrics from Firefox over prolonged repetitive usage. http://blargon7.com/2011/03/introducing-firefox-endurance-testing/

  22. BMW, just a matter of Seamonkey flipping the telemetry prefs.

    Dave, sure, should be straightforward. Just setup a local telemetry server and trigger a ping at the end of the test.

  23. I’ve got one question about the telemetry data: I enabled it and now looked at the “nsCycleCollector::Collect (ms)” (whatever that is, you should really try to make users know what you record in detail). It says: “29 samples, average = 14.8”, and this is my data (x:y): 4:-, 5:2, 6:5, 7:2, 8:9, 10:2, 14:3, 17:2, 20:1, 34:1, 40:1, 96:1, 114:-. Now, 5*2+6*5+…+96*1 = 412, whilst 2+5+…1 = 29. But 412/29 is 14.2069, not 14.8. What is wrong with the data? Is that a bug?

  24. ====
    ====
    You said: “we do not plan to tag reported data with unique identifiers. This will make it harder for us see how Firefox performance changes over time for particular users ”

    NO! Unless telemetry captures my formfill data, I want to make firefox better! Make it optional with the default “off”. And frankly,

    –faye kane, homeless brain
    More of my smartmouth at http://tinyurl.com/kanecave

  25. Hi Taras!

    I just installed Firefox last nightly (v7.0a1) and the about:telemetry extension isn’t compatible.
    Could you allow it to work on Firefox7 or do I have to install Add-on Compatibility Reporter to be able to use it ?
    (Or maybe this extension isn’t designed to work on Firefox 7 at all ?)

  26. In addition to what Alex noted, when I force the extension to run (Firefox 7.0a1, 64bbit version of 29 May on Windows, version of 30 May on Ubuntu) the about:telemetry frame appears blank. The HTML is fragmented, ending in the middle of the first style tag. (Included below removing 6 angle brackets.)

    htmlheadstyle type=”text/css”.main {float:left; border-style:solid; border-color:gray; border-width:1; white-space: nowrap; padding:10px }.bar DIV.above { background-color:

  27. Yura Zakalyuzhny

    Taras,

    the link to your extension is dead, please fix it.

  28. Some of my profiles point to http://telemetry.allizom.org, others to https://data.mozilla.com for the telemetry server. This what I get for multiple versions of Firefox and installing testing software.

    What is the correct server?

  29. @Yura, thanks updated link

  30. It was written before but I didn’t saw an answer so I’m asking again. Is it possible to get the option to include an anonymized-but-unique ID to allow for better data collection?