Testing the top 100 add-ons for memory leaks

Yesterday I mentioned that the plan for testing the top 100 add-ons for memory leaks had hit a snag — our list is that of the top 100 installed add-ons, rather than the top 100 enabled add-ons, and the latter is what we really want.

However, after some thought, I’ve concluded that there is likely to be a lot of overlap between the two lists.  For example, I’d be surprised if any of the top 50 enabled add-ons are not in the top 100 installed add-ons list.  Furthermore, this kind of testing will never give perfect coverage anyway.

Therefore, there’s little point in delaying the testing while we wait for a better top 100 list.  If you are interested in helping, please jump in.  And if you contact me privately I’ll be able to suggest some add-ons that are towards the more popular end of the top 100 list.  Thanks, and apologies for any confusion I have caused!

23 Responses to Testing the top 100 add-ons for memory leaks

  1. This is awesome… while you’re at it, can you test page load times? I know the focus of memshrink is memory usage but load times are also an important perf metric.

    Let me know how I can help, also.

    Cheng.

    PS: if you want something else to consider, see if you can get a regional breakdown of add-on usage. At least the top Russian add-ons tend to be different than the top US ones and should probably be tested on Russian sites.

    • Nicholas Nethercote

      Page loading is outside the scope of the testing. It’s much harder to test. The nice thing about zombie compartments is there are no shades of grey — either you have zombie compartment(s) or you don’t. Having said that, if some other aspect of performance is noticeably bad under an add-on the tester can certainly report that.

      As for how you can help — take a look at the etherpad linked in the bug, pick an add-on, and follow the testing instructions! :)

  2. I love seeing update messages like this:

    https://lastpass.com/upgrade.php?ver=1.90.4&type=ff&upgrade=true

    “Resolved: Firefox single immortal zombie killed.”

    Word is definitely getting out!

  3. Best line ever. So much so that I read the wiki for further illumination.

    Zombie compartments

    Sometimes, due to bugs in Firefox and/or add-ons, compartments are created that are never destroyed. These are a particular kind of memory leak, and they cause Firefox’s memory usage to increase gradually over time, slowing it down and making it more likely to crash.

    https://developer.mozilla.org/en/Zombie_compartments

    Ah, so that explains it. Fx’s been crashing a lot lately (since upgrading to 10) – maybe mortalizing the immortal zombie will make it stop.

  4. Dear Mr. Nethercote, my machine is an old computer with an 800 MHz Duron and 384 MB main memory running Windows XP (SP3). I have both Fx 3.6.27 and Fx 10.0.2 (the Tor bundle) installed. Even with a dozen add-ons to 3.6.27 and only three to 10.0.2, Fx 3.6.27 is more responsive and uses much less main memory. So, I am leery to scrap it in favor of the “new generation”.

    Will there come a day when Fx 11… or 12… or 13 matches 3.6.27 in terms of responsiveness and memory footprint?

    • Nicholas Nethercote

      There was a report that the add-ons included in the Tor bundle causes leaks, though others were unable to reproduce. See https://bugzilla.mozilla.org/show_bug.cgi?id=719329.

      More generally, are the three add-ons you have installed under 10.0.2 all also installed under 3.6.27? Comparing a vanilla Firefox 10.0.2 to a vanilla Firefox 3.6.27 would be a fairer comparison. Or if they both had the same add-ons installed.

  5. Thank you for your reply. Well, my Fx 3.6.27 and my Tor-bundle Fx 10.0.2 have in common NoScript, HTTPSEverywhere, and AdBlock. The 3.6.27 browser has 8 add-ons in addition, plus a number of plugins including Flash, Java, Acrobat… but still uses less memory. (I am counting only Firefox’s share of main memory in each case, not the memory consumption by Tor and Vidalia, the other components in the Tor bundle.)

    It just occurred to me that perhaps the Tor project people tweaked the Fx settings so that it keeps everything in main memory instead of swapping to hard disk? This might explain it.

    So what is the official Mozilla position on the respective memory usage of 3.6.27 versus the “new generation”?

    • Nicholas Nethercote

      There’s no official Mozilla position.

      My personal experience is “memory usage can vary enormously depending on your workload”.

      Ignoring the memory usage figures for a minute: do you see worse performance or more crashes with Firefox 10 than Firefox 3.6?

      • Performance/crashes in Fx 10 v. Fx 3.6:

        I don’t think either version has outright crashed on me yet. So please count that as a testimonial in favor of the hard work done by your QA people.

        Re performance, it is my impression that 10 is more sluggish, in addition to being a memory hog (by comparison to 3.6).

        What I probably should do is download 10.0.2 from mozilla.com (not from torproject.org) and compare the two versions on an even footing (each in “factory” condition).

        If you have a set of websites that you use for testing, I would be happy to try each one out and record loading times with a stopwatch and then report back to you.

        • Nicholas Nethercote

          Comparing the mozilla.com version of 10 is a good idea. We use the Alexa top 100 sites for some automated internal testing, but I suggest you instead measure sites that you visit a lot.

          BTW, I appreciate that you’re thinking about how to do fair comparisons. That’s more than 99.9% of users do! :)

  6. I had a old Pentium 3, that struggled with 3.6 and was unusable with Firefox 4 and I on, I put it out to pastures a while ago, and knowing how to build desktops it was relatively cheap and easy to scrap together a relatively modern replacement.

    With that said, I don’t think the current Firefox version can ever match Firefox 3.6 memory efficiency. There are too many changes under the hood. Also the memshrink project deals more with “leaks” as opposed to lowering Firefox’s baseline memory requirement. The “leaks” can be fixed, but the memory requirement for Firefox will only grow when more features are incrementally added.

    • Many of the early memshrink fixes were about reducing baseline memory use. That low hanging fruit is mostly done now. Stomping on leaks is harder work; but is the only major category of improvements left. Even there, most of the problem is now in add-ons; not the core browser.

      • Nicholas Nethercote

        You’re right that badly-written add-ons are definitely a big problem, possibly the single biggest problem.

        However, there’s plenty of improvements still to be made on baseline memory usage. Generational GC should be a big win for the JS engine. Compiling JS code to bytecode only when it is run could be another big win. Better heuristics for the decoding of images in the foreground tab will make a huge difference for image-heavy pages. Etc, etc :)

    • Nicholas Nethercote

      MemShrink deals with both “leaks” and baseline memory usage; see the introduction of https://wiki.mozilla.org/Performance/MemShrink.

      I think that memory consumption can be better than 3.6. I’ve heard anecdotal evidence that some people already find it to be better. And I think there’s still a lot of room for improvement. I don’t think we’ll ever reach a point where we can clearly say “version N is better than 3.6″ because it varies so much with workload and we’re not measuring 3.6 any more. My goal is for each version N to be better than version N-1.

  7. You have to remember that the internet Firefox 11 is dealing with is different to that which Firefox 3.6 dealt with. If you look at Tomshardware

    Whilst it may be shouted down by certain critics, the one thing that Tom’s Hardware does, is compare browsers on a truly level playing field. They’re all on the same PC, with the same hardware and software, put through the same tests on the same day etc etc.

    This is one of the few times where you get to see exactly how a truly add-on free Firefox 10.0.1 performs against Chrome 17 on a truly level playing field.

    When people on forums try and argue and compare performance, the number of variables that are likely to be different between the laptop of Hank in Colorado and the desktop of Pierre in Paris means that there is no comparison at all. It’s about as far from a controlled test environment as you could ever get.

    As far as memory performance is concerned Firefox 10 is only just behind Chrome 17 on the memory management, but on the memory usage, with both a single tab and 40 tabs, it beats Chrome pretty convincingly. Click on the link below:

    http://www.tomshardware.com/reviews/chrome-17-firefox-10-ubuntu,3129-14.html

    With Snappy fixes in place from Firefox 12 onwards, the performance gap will be closed more and more.

    • Nicholas Nethercote

      I personally think the Tom’s Hardware memory comparisons are crappy and don’t tell us much. Doing meaningful memory comparisons of that sort is *really* difficult.

      I’d prefer it if they did something like re-run their performance benchmarks on a machine with a small amount of RAM; I think that would be more meaningful than what they currently do.

  8. Saint Wesonga

    I believe the section on “Proactive testing of addons” at https://developer.mozilla.org/en/Zombie_Compartments mistakenly swaps Firefox 13 procedure with the one for 12 & earlier (which don’t have about:compartments).

  9. Memory leak and Add-on

    Despite the latest memory bug improvements in FF versions, I recently got a nasty one. After some tedious research time I found that WOT in my Google Search page triggered it. More specifically, some “Wot Safe Search Refinement [x]” option that, if turned off, ended the memory clog.
    So, if someone figure a cyclic sawtooth memory usage behavior in the Firefox process, check this item.

    Hope this could spare you a bunch of work, FF fellow.
    Cheers


    Intel T5600, XP SP3, FF 11 and a lot of add-ons

    • Nicholas Nethercote

      Thanks for the report. I just tested WOT as part of our top 100 add-on testing. I was unable to reproduce any zombie compartments. The steps I took and the results are in https://etherpad.mozilla.org/hSOVNzKZen near the bottom (add-ons are listed in alphabetical order).

      Or maybe the problem wasn’t a zombie compartment? Can you give precise steps to reproduce the problem and the exact symptoms? Thanks!

  10. Hi, Nicholas

    Yes, one of the consequences is the zombie comportment. The cause is excessive use of the free memory, leaving almost none.
    Since there is a huge diversity of combinations in hardware, software, add-ons etc, maybe this can’t always be reproduced. I, nowadays, have no access to other hardware, so…
    1. Install the WOT add-on
    2. Reopen Firefox and go to a Google search page
    3. Check if there is a “Wot Safe Search Refinement [x]” line, just bellow the search box. It’s the WOT add.
    4. Open Process Explorer
    5. Open the Firefox process properties window
    6. Select the “Performance Graph” tab

    If you are “lucky”, in no time you’ll see a constant and increasing use of the free memory by the FF process, until it suddenly shrinks. After a little while, the behavior star all over again, drawing a well defined growing sawteeth shape pattern. Eventually, Firefox crashes.
    If you get back to Firefox and disable the “Wot Safe Search Refinement [x]”, the strange memory behavior ends.

    As you can imagine it was very, very annoying and demanded much time to catch, so I thought on share and save others’ time.

    Hope this helps.
    Cheers


    Intel T5600, 2GiB
    XP32/SP3
    Zillions of programs and drivers
    Firefox 11.0
    WOT 20120302 add-on and a great bunch of other ones

    • Nicholas Nethercote

      I just tried and failed again to reproduce this, this time on my Windows 7 machine. I see a “Wot Safe Search [x]” line, but it’s missing the word “Refinement”, I wonder if that’s relevant. I’m just using a vanilla install of WOT, without having changed any settings.

  11. Denis Sylvestre

    I had the exact same problem in FF 12.0 with WOT 20120302, also on Google pages.