01
Jul 11

Telemetry Status

Telemetry Present

Telemetry infrastructure has only been deployed for week, but we are already gathering interesting data:

  • Memory usage from about:memory gives us an idea of what constitutes typical memory usage for Firefox
  • Cycle-collection overhead, stats tell us about browser pauses due to memory cleanup
  • Detailed startup profiling tells us whether our new library preloading logic is effective
  • Info on whether the browser was shutdown correctly will help us diagnose shutdown problems
  • Plugin enumeration timing to make sure my faster plugin enumeration stays fast
  • HTTP connection profiling to help optimize page loads

Since the branch point for Aurora is approaching in less than a week, I don’t expect many more probes for Firefox 7.

Telemetry Future

One problem with doing awesome optimizations that as code changes, they frequently get accidentally undone. I plan to add telemetry to keep tabs on every significant optimization that I do in the future, in addition to retroactively adding it to the past ones. I expect other Mozilla developers to do the same.
In addition to keeping tabs on Firefox performance, we can also learn about JavaScript feature adoption, the kind of hardware (and OS) that users run Firefox on, etc to better match users’ needs.


27
Jun 11

Telemetry is on in Nightly Firefox builds

Telemetry went live in Firefox Nightly builds over the weekend. Everyone who wants to contribute to making Firefox better now has an easy new way: opt-in to telemetry.
There are two ways to opt in:
A) Click yes when prompted to report performance
B) Enable it by going to Options/Preferences, then Advanced/General tab

You can check on the data collected by installing my about:telemetry extension.

 


22
Jun 11

Developers: How To Submit Telemetry Data

Telemetry is a way to gather stats about Firefox. Currently histograms are the main mechanism by which to gather data. There are 3 histogram types currently supported: exponential, linear and boolean. Exponential+linear histograms can accumulate numbers between 0 and a user-defined integer maximum, they differ in bucket size increments. Boolean histograms are meant to store 0 or 1.

Steps to Add a New Metric

1) Add a histogram definition to TelemetryHistograms.h specifying a histogram id, parameters and a description. For example HISTOGRAM(MYHGRAM, 1, 10000, 50, EXPONENTIAL, “Time (ms) taken by shiny new metric”) defines a MYHGRAM histogram with a minimum value of 1, maximum 10000 and 50 buckets that grow exponentially with the description specifying units as milliseconds.
2a) C++:

#include <mozilla/Telemetry.h>

….
Telemetry::Accumulate(Telemetry::MYHGRAM, <some interestin integer value goes here>)

2b) JS:

Telemetry = Cc["@mozilla.org/base/telemetry;1"].getService(Ci.nsITelemetry)
var h = Telemetry.getHistogramById("MYHGRAM");
h.Add(<some interesting value goes here>);

3) Install about:telemetry addon, go to about:telemetry to check that the chosen histogram type and parameters summarize your data in a useful way.

4) Commit and wait for results to come in.
FAQ
Q: What about those UMA_* macros?
A: Don’t use them. I added TelemetryHistograms.h so telemetry can be easily reviewed by security and privacy police. It also avoids a few pitfalls such as accidentally initializing the same histogram with different parameters, etc.

Q: When will telemetry be deployed?
A: We will land the UI to turn it on as soon as an updated privacy policy is posted. It should show up in nightly builds by the end of the week.

Q: How do I access the gathered telemetry data?
A: TBD. Data is stored on metrics server, we will figure this out soon.

Q: What about addons?
A: TBD. At the moment addons are free to inspect telemetry data in their browser. We haven’t decided on a process to let addon authors add new probes and access stats. For now, addons should not participate in telemetry.


01
Jun 11

Telemetry Updates

My previous post was too optimistic. There will be no telemetry in Firefox 6. Due to the multitude of reviews involved we slipped and are now aiming for Firefox 7. Bug 659396 tracks various ongoing telemetry tasks.

I updated my about:telemetry extension to work with Firefox 7 nightlies. Additionally, my friend, David, helped me apply some nasty CSS tricks to make the histograms look like histograms. I’m open to further CSS contributions. I haven’t listed the extension on AMO because we plan to have this functionality integrated into Firefox soon (hopefully 7).

To turn on telemetry,we have to: a) finish up the telemetry opt-in UI and b) update our privacy policy.

Thanks to everybody who manually flipped the pref to turn on telemetry. Having early feedback on this feature is awesome.


13
May 11

Firefox Telemetry

Benchmarks Suck

Mozilla has traditionally relied on [Talos, Sunspider, Kraken, etc] benchmarks to optimize Firefox. Unfortunately there are two problems with benchmarks: a) it is hard to write good benchmarks (see all of the complaints about Sunspider) b) the most perfect synthetic benchmarks do not completely correspond to actual user usage. Firefox with a well-used profile, anti-viral software, well-aged Windows and 30 addons will not perform the same as it does in our clean benchmarking environment.

For my team this became obvious in Firefox 4 once we started recording Firefox startup times. Turned out that it is easier to work on fixing startup performance than make our synthetic test closely reflect real world startup speed.

Telemetry

There is only one solution to this problem: develop telemetry infrastructure to measure Firefox performance in the wild. Beginning with version 6, Firefox will ask users to opt-in to sending anonymous usage statistics about performance, user interface feature usage, memory usage, and responsiveness to Mozilla. This information will help us improve future versions of Firefox to better fit actual usage patterns.

This functionality is already present in our major competitors. Unlike our competition we do not plan to tag reported data with unique identifiers. This will make it harder for us see how Firefox performance changes over time for particular users (or easily tell whether some users are disproportionately represented due to sending more reports). We take our users’ privacy seriously, so this seems like a reasonable trade off.

Yesterday I landed the reporting part of telemetry (bug 585196). We are still working on UI, official server-side and on updating the privacy policy.

Above screenshot shows some of the data that will be gathered for users that opt-in to telemetry.

Please help us get a headstart on telemetry. In the recent nightlies, go to about:config and set toolkit.telemetry.enabled to true. Once the pref is set, Firefox will send interesting performance data to the telemetry test server. The metrics are very compact and are sent out no more than once a day.

Since there is no UI yet, install my about:telemetry extension and navigate to “about:telemetry” to see the metrics collected.


14
Apr 11

Arguing about startup: Communicating performance

Recently the addon team started working towards penalizing addons that penalize our startup. We have solid data shows that startup gets worse as more addons installed so the effort is justified. Justin published this picture to illustrate the problem:

However some technical mistakes were made. Wladimir (AdBlock+ guy!) has been busy exposing them on his blog. Thanks Wladimir! I spent a couple of years understanding Firefox startup, so I really appreciate Wladimir’s remarkable speed/quality in poking holes in AMO approach.

Rating Addon Performance

Wlad’s latest point is regarding how addon impact is measured on warm startup with an empty profile. I agree that addon impact on warm startup with clean profile is going to be different from that of cold startup with a dirty real-world profile. I also agree that it is weird to primarily measure warm startup given that our data clearly indicates that most users are experiencing cold startup.

However startup is an irritating multidimensional problem influenced by a large number of factors. Should we suck it up and let addons kill warm startup (and risk ruining firefox upgrade times, pissing off web developers)? How does one choose a typical dirty profile? Creating a “typical” dirty profile is a tough problem (ie due to privacy, disk fragmentation) and there is no reason to let clean profile performance be degraded.

So an addon performance rating will not be perfect. Time added by addon during warm startup is the easiest starting point and is one of the more deterministic measures (Wlad’s blog says otherwise, but measuring other startup kinds have even more noise). This is no worse than choosing browsers based on their JavaScript benchmark performance.

I think a much awesomer addon performance rating may be obtainable by clever statistics boffins by analyzing our aggregated startup data. Perhaps the Mozilla Metrics will make it a reality, but in the meantime we’ll have to make do with an imperfect approach.

Coming soon: Why is the measurement approach in Jorge’s post is overly complicated and somewhat incorrect.


10
Mar 11

“Start Faster” Addon

A large proportion of our startup time is spent on loading the Firefox library(.dll, etc) files. This is true on all of our platforms. In my previous post I thought I discovered a way to load the Firefox binaries more efficiently on Windows(bug 627591). Further testing revealed that to not be true in all cases and that the Windows Prefetch service was still killing our startup speed. Microsoft does not seem to provide a way to opt out of the Windows Prefetch ‘service’.

There is a DIY way to opt out of prefetch by gaining Administrator privileges and deleting files out of the prefetch directory. Our is plan is to provide a Windows service to handle Firefox updates (bug 481815). Additionally the service would be able to do useful things like delete prefetch files, defragment Firefox databases(or at least help report on fragmentation levels). This is all tricky stuff.

In the meantime, to test my theory I wrote a test extension. This is a restartless extension that adds a Windows service and a wrapper executable to significantly speed up Firefox startup after rebooting (oh the irony!). After installing the extension, “Faster Firefox” shortcut on the Desktop should result in up to 2x speed up in Firefox startup. This addon is a rough proof of concept to play with while I bake this functionality into Firefox. Comments, improvements are welcome. Note that launching Firefox by shortcuts other “Faster Firefox” is slower while this extension is installed (this will be fixed once preloading functionality is integrated into Firefox properly). This addon does not yet work on XP because I do not yet have an XP enviroment to test/develop in (patches are welcome).


14
Jan 11

Builtin Startup Measurement

I got used to measuring startup the complicated way (example here). It’s complicated enough that many people prefer to use stopwatches.

Turns out modern operating systems can help applications self-diagnose startup speed. Thanks to landing bug 522375 we now provide an API for measuring startup speed. For example, now I know that xpcshell takes forever to startup on mac

./xpcshell -e 'print(new Date() - Components.classes["@mozilla.org/toolkit/app-startup;1"].getService(Components.interfaces.nsIAppStartup_MOZILLA_2_0).getStartupInfo().process)'

At any point one can now go to the error console and type in

uneval(Components.classes["@mozilla.org/toolkit/app-startup;1"].getService(Components.interfaces.nsIAppStartup_MOZILLA_2_0).getStartupInfo())

or

var si=Components.classes["@mozilla.org/toolkit/app-startup;1"].getService(Components.interfaces.nsIAppStartup_MOZILLA_2_0).getStartupInfo(); si.sessionRestored-si.process

to get various interesting timestamps. So next time you see a startup take a surprising amount of time, you can go and poke around to see where that time was spent. At the moment there are 4 datapoints:

  1. .process – Process creation timestamp. This is cool because this happens before any library code is executed.
  2. .main – XRE_main timestamp. My favourite thing to do is to subtract .process from .main. This demonstrates huge overheads that many application programmers refuse to believe in.
  3. .firstPaint – Timestamp of the first intended paint. This coincides with when the user sees the first sign of life.
  4. .sessionRestore – Timestamp of session restore, ie when the browser becomes useful.

Lots of people helped in getting this feature landed, but two people stand out. Daniel Brooks originally figured how to expose Windows/Linux startup speed in Mozilla. Mike Hommey devised a morally-reprehensible way to get precise startup speed out of the idiotic way that Linux presents it.

Linux Sucks

Mac and Windows expose process creation times via human time. It’s just like any other API with time in it. Linux provides process startup speed in imbecile jiffies-since-boot. That’d be irritating enough, but to piss people off further there is no way to convert that into human time. Clever ps developers resort to calculating jiffies/second by comparing against seconds-since-start in /proc/uptime. Unfortunately that does not even come close to providing anything close to millisecond resolution needed for useful numbers. Mike’s idea of timing startup of another thread/task to get a known jiffy-stamp got us precise-enough numbers (around 10ms resolution with most kernels, see patch for details). At least Linus was kind enough to let mere user-space devs obtain the current tick-rate.

Update:

Linux doesn’t suck anymore, a commenter pointed me to a relatively new(2006) taskstats interface which appears to work sanely like the BSD one.

Mike Hommey made an about:startup extension using this API.


30
Nov 10

Crapware and Firefox

I completely agree with Asa that having unwanted crap forced upon the user is morally wrong. We should do a better job of undoing this kind of braindamage. In the meantime here is a brief rant on the parasitic underpinnings of crapware.

Until recently, I have been testing Firefox on my own installs of Windows. I had no idea how aggressive bundleware could be. Then I got this piece-of-crap i7 Acer laptop with Windows 7 (and relatively little crapware preinstalled) and tried to use it as my primary machine. Suddenly, I could reproduce a lot more “slow” scenarios. I even went further and tried installing common crapware known as AVG to reproduce more bugs.

Turns out almost every vendor tries to mix in crap into Firefox. Acer, Microsoft Office/Silverlight, Adobe flash/acrobat, Google, AVG, etc all added unwanted functionality to my Firefox. I marveled at all kinds of “helpful” functionality such as the wonderful ability to click on a link in a webpage and have Google chrome install without any warning that the webpage is about to execute a windows program. AVG adds a couple of extensions that make Firefox start up 0.5-4x slower.

So far I noticed 2 vectors of attack: plugins and extensions. Plugins are fun because those get added by registering bonus plugin directories. Plugin directories are usually just application directories that contain plugins. This means Firefox gets to slowly rummage through bonus application directories looking for what might be a plugin. Extensions are fun because unlike plugins (which affect most browsers on the computer), extensions are very browser-specific. Most extension crapware doesn’t yet support Chrome. Installing things like AVG retards Firefox performance while Chrome escapes unmolested.

Benjamin, I don’t think these software vendors are “doing exactly as we ask of them.”

Personally, I would like us to be a lot more aggressive about blacklisting ill-performing software. Ie we need to go above and beyond warning users when crapware. I would like to to actively check performance of popular plugins/addons and ban them if they are substandard.


16
Nov 10

Performance Update. Fragmentation: Mostly fixed. GCC: work-in-progress.

Fragmentation: SQLite & Friends

I am happy to report that the SQLite fragmentation problem is now solved. I copied my profile a month ago, and my places.sqlite is still in a single fragment! There was a similar fix done to Firefox disk cache. Thanks to helpful comments on my OSX preallocation cry for help, we now preallocate efficiently on OSX too.

Startup cache is the last remaining bastion of fragmentation, but that’s already 10x better than it was a month ago. I have two complimentary solutions for that: either omnijar startup cache generation for core code and/or write the cache more efficiently.

Firefox 4 will be a lot more gentle on those spinning platters.

GCC

I helped Jan Hubicka on a GCC summit paper. Those nasty static initializers will not be a hassle in GCC 4.6!

I keep wanting to blog about how we switched to GCC 4.5 and how life is wonderful, but life didn’t work out this way. So far we tried switching away from 4.3 compiler three times. The first time GCC completely failed in terms of -Os performance. C++ -Os is more bloated in 4.5 (because that option is benchmarked on C apps). Then it turned out that libffi was being miscompiled (we also found a related bug in libffi). Last time, we tried switching to GCC 4.5 + -O3 since that performs much better than -Os, but that broke sunspider. Hopefully we can fix the sunspider issue and try again next week. I would really like to utilize GCC PGO to produce fastest possible Linux Firefox builds.

Nonetheless I happy with recent GCC progress. With Jan’s help, GCC will eventually be very good at compiling Mozilla. In my spare cycles I’ve been working on setting up GCC benchmarks using Mozilla to help avoid future surprises like we discovered in GCC 4.5. More on this later.