My daughter Keira turns four in a few months. She’s been practising reading the letters on my Firefox hoodie. On Saturday, we started with the front.
Keira: F – I – R – E – F – O – X
Me: What does that spell?
Keira: Firefox!
Then we moved to the back.
Keira: M – O – Z – I – L – L – A
Me: What does that spell?
Keira: <pause> Vanilla.
It’s clearly time to rename “add-ons” as “crush-ins”.
Leaky add-ons are a big problem, but there’s been lots of action relating to them recently. What follows is a summary of the situation and a request for help from currently Nightly testers who use add-ons.
Kyle Huey’s patch to prevent Chrome-to-Content leaks landed recently. In theory it would prevent almost all add-ons zombie compartments, which constitute the majority of leaks from add-ons. And in practice, it appears to be working splendidly.
I did some testing of eight add-ons that are known to leak: McAfee SiteAdvisor 3.4.1(the old version that leaked every content compartment), Firebug 1.9.1, PicPasso 1.0.1, LoL 1.5, YouTube Ratings Preview 1.03, Enter Selects 7, HttpFox 0.8.10, and Link Resolve 0.7. I tested with a “before” browser build that lacked Kyle’s patch, and an “after” browser build that had it. For every one of the tested add-ons, I could reproduce the zombie compartment(s) with the “before” build, but could not reproduce any zombie compartment(s) with the “after” build. In other words, the patch worked perfectly.
To understand how important this is, consider this comment on Kyle’s blog describing the effect of the patch.
I seem to be seeing really significant improvements from this. Normally Fx would start at 170MB and by the end of the day that would often be creeping up to 800 or 900 MB. Today using latest nightly I again started at 170, but now at the end of the day I’m only at 230MB!
That’s a 4x reduction in memory consumption.
What effect does this have? The obvious benefit of reduced memory consumption is that there will be less potential for paging. For example, if the above user is on a low-end machine without much RAM, a reduction in memory consumption from 800+MB to 230MB might greatly reduce paging, which would make Firefox much faster.
However, what if the user has a high-end machine with 16GB of RAM? Then paging isn’t an issue. But this improvement will still be a big deal on such a machine. This is because garbage collection and cycle collection cause pauses, and the length of the pauses are roughly proportional to the amount of live heap memory. (Incremental garbage collection will soon be enabled, which will result in smaller garbage collection pauses, but there are no plans for incremental cycle collection and so cycle collection pauses will still be relevant.) So even on high-end machines with lots of RAM, leaks can greatly hurt browser performance. In other words, add-on leaks are a Snappy issue just as much as they are a MemShrink issue.
[Update: Kyle found a simple way to fix this problem with the Add-on SDK.]
So Kyle’s patch is great, but the cutting of the chrome-to-content references was unlikely to be totally free of side-effects. And sure enough, last week Josh Matthews noticed that Dietrich Ayala’s Wallflower add-on was causing many zombie compartments. Hmm, wasn’t Kyle’s patch supposed to stop exactly that? Well, it was expected to stop the most common cases, but there are some other leaks that could still cause zombie compartments.
The weird thing with this leak is that Wallflower is a really simple add-on. It’s built with the Add-on SDK and is only a few lines of code. It turns out that Kyle’s patch caused the Add-on SDK to leak badly. The exact reason is complex; you can see Kyle’s explanation here.
This sounds really bad. Lots of add-ons are built with the Add-on SDK. Have we just exchanged one class of add-on leaks for another? I don’t believe so, for two reasons.
Still, the situation is far from ideal, because lots of add-ons are still built with older SDK versions. So, what to do? There’s a bug open discussing this question, but it’s gotten bogged down due to differing opinions on what to do. Version 1.6.1 happened to fix all leaks in the Add-on SDK that were known prior to this problem, so we should be encouraging add-on authors to rebuild with it anyway.
In the meantime, if you use Nightly builds and use add-ons, please monitor about:compartments and file bugs if you find that any add-ons are causing zombie compartments. If Wallflower is representative, the leaks will be very bad and so shouldn’t be hard to spot. Firefox 15 is scheduled for release on August 28th; we need as many affected add-ons to be rebuilt with the latest SDK before that date to minimize potential problems.
Another cause of high memory consumption in add-ons is FUEL. I know very little about it other than it’s also part of the Add-on SDK sort of a proto-version of the Add-on SDK. The details are here and here. This is causing bad performance in Readability and probably other add-ons. Something to be aware of.
The big news this week is that Kyle Huey made chrome-to-content leaks impossible. Kyle wrote about this in some detail. This particular kind of leak can happen in Firefox, but more importantly, it’s responsible for most of the leaks in add-ons that we know about. In other words, it should greatly reduce the problem of leaky add-ons, a problem I previously identified as the #1 MemShrink issue. It’ll take a while to see the exact effects and whether they match expectations. Nonetheless, this is a big deal!
Olli Pettay fixed a leak relating to Geolocation.
The following add-ons had leaks fixed: Collusion, SearchMenu, panchang, HTML5 Extension for Windows Media Player Firefox Plug-in.
I reduced the size of JSScripts (here, here and here). This increased the number of JSScripts that fit in a GC arena from 31 to 42 (on 32-bit platforms) and from 19 to 28 (on 64-bit platforms). This reduced the size of the “scripts” entry in about:memory — which is often multiple MB for the compartments of complex web sites and apps like Gmail — by approximately 30%. (The final two patches in the sequence are being held up by the current tree closure, but will land as soon as that has passed.)
Nathan Froyd reduced the size of mozilla::dom::Link — one of which exists for every <a> or <link> or <area> element shown in a page — by 4 bytes (on 32-bit platforms) and 8 bytes (on 64-bit platforms).
Ehsan Akhgari reduced the size of nsEditor — one of which exists for each text area in a page, and some other places — by 16 bytes on 64-bit platforms.
PC Advisor featured an article about browser performance. The article criticized the standard browser benchmarks for (a) not representing everyday browsing and (b) underestimating the effects of memory consumption on browser performance — two things that I agree with.
The article also presented some cross-browser memory consumption results from real-world-ish workloads. The testing was imperfect — for example, it didn’t measure memory consumption after any kind of prolonged usage — but I liked the fact that the author had actually thought about things instead of just mindlessly running the standard benchmarks.
And with my PR hat on, I was happy with the conclusion.
Firefox is the clear winner of the bunch. It was the only browser that did not slow things down and I recommended it for both lower-end mobile devices and high-end desktops.
In general, cross-browser memory consumption comparisons are difficult to do well. My preferred method is to run some set of benchmarks on a machine with a small amount of RAM, as that gives a genuine indication of the performance effect. “Browser A was 50% slower on benchmark X when I halved the available RAM, but browser B was only 10% slower” is much more meaningful than “Browser A used 20% more memory on benchmark X than browser B”. Choosing good benchmarks is still not easy, though.
Here are the current bug counts.
I’m thinking about omitting the bug counts from future MemShrink reports. I don’t feel like they add much, but I’d be interested to hear if people think otherwise.
This blog recently moved from blog.mozilla.com/nnethercote to blog.mozilla.org/nnethercote. This move broke the captcha I use for comments for a couple of days. The breakage has now been fixed, sorry for any inconvenience!
As promised last time, this MemShrink report covers two weeks’ worth of activity.
This week Kyle Huey, Justin Lebar, Andrew McCreight, Jet Villegas flew to Melbourne to bunker down with me for a MemShrink work week. This is great for two reasons. First, it gives all of us an excuse/opportunity to ignore the usual myriad of attention-grabbing tasks and focus solely on MemShrink stuff for a week. (For example, Jet is the manager of the layout team but he’s managed to do a couple of days of coding!) Second, it lets us collaborate and pick each others’ brains much more easily and efficiently than normally.
The following add-ons had zombie compartments fixed.
I’ve said before that add-on leaks are the #1 cause of high memory consumption in Firefox. Numerous add-ons have had leaks fixed, but this is a slow process. However, Kyle has been making good progress this week on a patch that has the potential to mitigate the majority of add-on leaks that cause zombie compartments. It currently passes all tests on try server and is undergoing review. Watch this space!
Version 1.6 of the Add-on SDK was released, which fixed all remaining known leaks. Except that a new leak was found shortly before release! This was found too late to be fixed before the deadline for 1.6, but fortunately, it was quickly fixed in version 1.6.1. The Add-on SDK developers (especially Alexandre Poirot) have fixed a ton of leaks in the past few months, so it’s great to see this milestone reached.
Olli Pettay fixed a leak relating to event listeners.
Henrik Skupin released version 0.3 of the MemChaser add-on. It features improvements to the UI and the logging.
Jonathan Kew optimized the representation of font character maps. Depending on the platform, this can save around 1MB of memory, or more if you have lots of fonts installed.
Jared Wein landed an option for making plugin content click-to-play. This isn’t marked as a MemShrink bug but if a user can ignore such content, it has the potential to reduce memory consumption significantly in some cases.
Here are the current bug counts.
Lots of movement there. This is because we’ve been going through the P2 bugs and closed a lot of them because they were dups, or lacked enough information to do anything useful, or we decided they were no longer worthwhile. We also downgraded quite a few P2 bugs to P3. As a result, the P2 list is much more interesting than it was. And we’re only partway through it!
I mentioned last time that I would be writing these reports every two weeks from now on, instead of weekly. Similarly, we’ve decided to reduce the frequency of MemShrink meetings to once every two weeks. The next meeting will be on May 1st.
This change does not reflect a reduction in the amount of work being done on MemShrink bugs. Rather, it reflects a reduction in the number of new MemShrink bugs being filed. At the past few meetings we’ve had only a handful of new bugs each time and triage has only taken a few minutes. (In comparison, in the early days of MemShrink, several times we didn’t get through all the new bugs in an hour.)
This leaves us lots of time for talking about other things; in fact, more time than we’ve needed. And time spent in meetings is time not spent fixing problems, so this feels like the right thing to do.
John’s Schoenick’s areweslimyet.com (AWSY) has had its password removed and is now open to the public!
This is a major milestone for the MemShrink project. It shows the progress we have made (MemShrink started in earnest in June 2011) and will let us identify regressions more easily.
John’s done a wonderful job implementing the site. It has lots of functionality: there are multiple graphs, you can zoom in on parts of graphs to see more detail, and you can see revisions, dates and about:memory snapshots for individual runs.
John has also put in a great deal of work refining the methodology to the point where we believe it provides a reasonable facsimile of real-world browsing; please read the FAQ to understand exactly what is being measured. Many thanks also to Dave Hunt and the QA team for their work on the Mozmill Endurance Tests, which are at the core of AWSY’s testing.
Update: Hacker News has reported on this.
Frequent readers of this blog will be familiar with zombie compartments, which are JavaScript compartments that have leaked, due to defects in Firefox or add-ons. Windows (i.e. window objects) can also be leaked, and often defects that cause compartments leaks will cause window leaks as well.
Justin Lebar has introduced the notion of “ghost windows”. A ghost window is one that meets the following criteria.
The basic idea is that a ghost window has a high chance of representing a genuine leak, and this automated identification of suspicious windows will make leak detection simpler. Justin has added ghost window tracking to about:memory, about:compartments, and telemetry. (These three bugs were all marked as MemShrink:P1.) Ghost window tracking is mostly untested right now, but hopefully it will become another powerful tool for finding memory leaks.
We’ve been tracking leaky add-ons in Bugzilla for a while now, but we’ve never had a good product/component to put them in. David Lawrence, Byron Jones, Stormy Peters and I have together created a new “Add-ons” component under the “Tech Evangelism” product. The rationale for putting it under “Tech Evangelism” is that it nicely matches the existing meaning of that phrase — it’s a case where a non-Mozilla entity is writing defective code that interacts with Firefox and hurts users’ experiences with and perceptions of Firefox, and Mozilla can only inform, educate and encourage fixes in that defective code. This component is only intended for certain classes of common defects (such as leaks) that Mozilla contributors are tracking. It is not intended for vanilla add-on bugs; as now, they should be reported through whatever bug-reporting mechanism each add-on uses. I’ve updated the existing open bugs that track leaky add-ons to use this new component.
Leaks in the following add-ons were fixed: Video DownloadHelper (the 2nd most popular add-on on AMO!), Scrapbook Plus, Amazon Price Tracker.
This week’s bug counts:
Good progress on the P1 bugs!
Many of the weekly MemShrink reports lately have been brief. From now on I plan to write a report every two weeks. This will make things easier for me and will also ensure each report is packed full of interesting things. See you again in two weeks!
Jorge Villalobos encouraged add-on authors to check for memory leaks on the official Mozilla add-ons blog.
The following add-ons had memory leaks fixed: FavSync, Download Youtube Videos, Page Speed.
The follow add-ons were downgraded to “preliminary review” status on AMO, giving them less visibility: QuickDrag, Yahoo Toolbar. The Yahoo Toolbar case is interesting; it is available on AMO, but only has 47,618 users there. But the number of installed copies is much larger than this. (I can’t say how much larger, because the exact numbers are not public knowledge.) So it’s clear that most people who have it installed got it elsewhere, probably directly from Yahoo. Given that AMO accounts for only a tiny fraction of the installations, the effect of the downgrade in AMO review status is negligible. Furthermore, Jorge’s attempts to contact Yahoo appear to have failed. Finally, the only other means we have for addressing add-ons with problems is block-listing them, but that would be overkill in this case. So we are at an impasse. It’s a frustrating situation!
Tim Taubert fixed two leaks in the recently updated new tabs page.
Jonathan Kew added memory reporters for thebes font objects. With text-heavy pages open, these can account for 10s of MBs of memory.
Andrew McCreight fixed some shutdown leaks involving the XBL cache.
Igor Bukanov did something with JS principals that I won’t pretend to understand. This bug wasn’t marked with the MemShrink tag, but it gave good Talos results, reducing the number of malloc calls and the maximum heap size on several platforms.
This week’s bug counts:
Nothing spectacular there.
The past week was fairly quiet for MemShrink.
Our testing of the top 100 installed add-ons is progressing slowly. We need more help here. Please join in if you are interested!
The Image Zoom and Baow add-ons were downgraded to “preliminarily reviewed” status because they cause zombie compartments, as per the new AMO policy. The main effect of the downgrade is that they’ll show up lower in AMO search results.
I modified the storage of CSS properties and values to minimize waste caused by alignment. On 64-bit platforms this reduced the size of a property/value pair from 24 bytes to 18 bytes, which saves about 0.5MB for an instance of Gmail. (Gmail uses a lots of CSS!) On 32-bit platforms the reduction is 1/3 the size, with a property/value pair dropping from 12 to 10 bytes.
Justin Lebar capped the amount of memory used for decoded images. Decoded images in background tabs are currently discarded 20-40 seconds after they stop being visible in the foreground tab. With Justin’s patch, there is now a second criterion: if the decoded image cap is exceeded, decoded images in background tabs will be discarded until the cap is met. (Decoded images in the foreground tab are never discarded, so it’s possible that all decoded images in background tabs are discarded and the cap is still exceeded.) The cap defaults to 50MiB, but it can be changed in about:config with the image.mem.max_decoded_image_kb option. The main effect of this change is that decoded images in background tabs will be discarded more quickly than they used to be in some circumstances.
This week’s bug counts:
That’s a net reduction of five bugs! Furthermore, we only had five bugs to triage in today’s MemShrink meeting. Good signs.
Versions of McAfee’s SiteAdvisor add-on prior to 3.4.1.195 have been soft-blocked due to an extreme memory leak. This means that it will be disabled within Firefox, but users can re-enable it if they want to. According to McAfee, most users should have been upgraded to 3.4.1.195 by now. Unfortunately, version 3.4.1.195 still has some leaks, and progress on fixing them has stalled. (If SiteAdvisor were an AMO add-on there’s a good chance it would have been downgraded to “preliminarily reviewed” by now.)
Leaks in the Lastpass, Fireshot, and Amabay add-ons have been fixed.
Two weeks ago I mentioned that the not-yet-public areweslimyet.com detected a large regression in memory consumption caused by incremental garbage collection. Bill McCloskey adjusted the GC and CC heuristics in order to fix this.
Justin Lebar previously added monitoring of available physical and virtual memory monitoring on Windows; if either goes below a threshold Firefox attempts to free up memory. This week he added monitoring of available commit space.
When you switch to a new tab, decoded image data from the old tab is kept until a 20 second timer expires. This is a good idea because you might switch back to the original tab quickly. However, until this week, this don’t-discard-it-immediately behaviour was also used when a tab is closed! Justin Lebar landed a patch which separates these two cases, i.e. it discards decoded image data immediately when a tab is closed.
I tweaked the presentation of the “window-objects” sub-tree in about:memory so that the memory used within each browser tab is more obvious. This is another step towards true per-tab memory reporting. Here’s an example for a tab in which I navigated first to valgrind.org and then to www.mozilla.org; the URL within the “top(…)” line is the one shown in the address bar.
│ ├──1,933,824 B (02.68%) -- top(http://www.mozilla.org/, id=13) │ │ ├──1,497,480 B (02.08%) -- active │ │ │ ├──1,496,456 B (02.08%) -- window(http://www.mozilla.org/) │ │ │ │ ├────876,912 B (01.22%) -- layout │ │ │ │ │ ├──569,456 B (00.79%) ── arenas │ │ │ │ │ ├──238,224 B (00.33%) ── style-sets │ │ │ │ │ └───69,232 B (00.10%) ── text-runs │ │ │ │ ├────316,216 B (00.44%) ── style-sheets │ │ │ │ └────303,328 B (00.42%) ── dom [2] │ │ │ └──────1,024 B (00.00%) -- window([system]) │ │ │ └──1,024 B (00.00%) ── dom │ │ └────436,344 B (00.61%) -- cached │ │ └──436,344 B (00.61%) -- window(http://valgrind.org/) │ │ ├──312,496 B (00.43%) -- layout │ │ │ ├──170,448 B (00.24%) ── style-sets │ │ │ ├──125,040 B (00.17%) ── arenas │ │ │ └───17,008 B (00.02%) ── text-runs │ │ ├───76,600 B (00.11%) ── dom │ │ └───47,248 B (00.07%) ── style-sheets
This week’s bug counts:
We only had to triage five bugs in today’s MemShrink meeting. I don’t remember ever having such a small number. As a result, we spent some time reviewing the P1 bugs, and decided that several could be downgraded to P2 because the situation had improved in some fashion.
Yesterday I mentioned that the plan for testing the top 100 add-ons for memory leaks had hit a snag — our list is that of the top 100 installed add-ons, rather than the top 100 enabled add-ons, and the latter is what we really want.
However, after some thought, I’ve concluded that there is likely to be a lot of overlap between the two lists. For example, I’d be surprised if any of the top 50 enabled add-ons are not in the top 100 installed add-ons list. Furthermore, this kind of testing will never give perfect coverage anyway.
Therefore, there’s little point in delaying the testing while we wait for a better top 100 list. If you are interested in helping, please jump in. And if you contact me privately I’ll be able to suggest some add-ons that are towards the more popular end of the top 100 list. Thanks, and apologies for any confusion I have caused!