Week 53-54? We’ve been going for over a year! To celebrate MemShrink’s 1st birthday, I wrote about the major achievements we’ve made in the past year.
Memory Reporting
I overhauled the cross-cutting “Other Measurements” shown for JavaScript in about:memory. It’s now in two parts. The first part is a functional breakdown of the memory consumption of the main JavaScript runtime, i.e. all JavaScript memory usage outside of web worker threads. Here’s an example.
194,179,064 B (100.0%) -- js-main-runtime ├──159,064,360 B (81.92%) -- compartments │ ├───73,850,880 B (38.03%) -- gc-heap │ │ ├──22,014,080 B (11.34%) -- objects │ │ │ ├──14,119,616 B (07.27%) ── non-function │ │ │ └───7,894,464 B (04.07%) ── function │ │ ├──19,553,168 B (10.07%) ── unused-gc-things │ │ ├──14,014,208 B (07.22%) -- shapes │ │ │ ├───6,747,240 B (03.47%) ── tree │ │ │ ├───4,926,000 B (02.54%) ── dict │ │ │ └───2,340,968 B (01.21%) ── base │ │ ├──10,204,960 B (05.26%) ── strings │ │ ├───5,338,816 B (02.75%) ── scripts │ │ ├───1,493,472 B (00.77%) ── type-objects │ │ ├───1,229,456 B (00.63%) ── arena-admin │ │ └───────2,720 B (00.00%) ── sundries │ ├───29,459,680 B (15.17%) ── analysis-temporary │ ├───14,484,704 B (07.46%) ── script-data │ ├───13,062,744 B (06.73%) ── string-chars │ ├────9,300,704 B (04.79%) -- shapes-extra │ │ ├──3,325,952 B (01.71%) ── compartment-tables │ │ ├──2,845,984 B (01.47%) ── tree-tables │ │ ├──2,125,056 B (01.09%) ── dict-tables │ │ └──1,003,712 B (00.52%) ── tree-shape-kids │ ├────7,688,912 B (03.96%) -- type-inference │ │ ├──5,750,576 B (02.96%) ── script-main │ │ ├──1,368,560 B (00.70%) ── object-main │ │ └────569,776 B (00.29%) ── tables │ ├────6,232,272 B (03.21%) -- objects │ │ ├──4,406,720 B (02.27%) ── slots │ │ ├──1,726,640 B (00.89%) ── elements │ │ └─────98,912 B (00.05%) ── misc │ ├────4,256,768 B (02.19%) ── cross-compartment-wrappers │ └──────727,696 B (00.37%) ── mjit-data ├───19,472,384 B (10.03%) -- gc-heap │ ├──18,006,016 B (09.27%) ── unused-arenas │ ├───1,458,176 B (00.75%) ── chunk-admin │ ├───────8,192 B (00.00%) ── decommitted-arenas │ └───────────0 B (00.00%) ── unused-chunks └───15,642,320 B (08.06%) ── runtime
The contents of the “js-main-runtime/compartments” sub-tree is the sum of all the individual compartment entries in the “explicit/js” sub-tree, and the “js-main-runtime” total is the same as the “explicit/js” total. This new view will become more important when per-tab memory reporting is implemented which will cause each tab’s JavaScript memory consumption to be reported separately.
The second part is designed to show the utilization of the garbage-collected heap. The new measurements and presentation are more detailed and easier to understand than the old version. Here’s an example.
93,315,072 B (100.0%) -- js-main-runtime-gc-heap-committed ├──55,755,888 B (59.75%) -- used │ ├──53,068,256 B (56.87%) ── gc-things │ ├───1,458,176 B (01.56%) ── chunk-admin │ └───1,229,456 B (01.32%) ── arena-admin └──37,559,184 B (40.25%) -- unused ├──19,553,168 B (20.95%) ── gc-things ├──18,006,016 B (19.30%) ── arenas └───────────0 B (00.00%) ── chunks
One thing this makes clear is that our “unused” fraction is often very high. We’ll need a compacting generation garbage collector to really fix that. This is being actively worked on, but it’s a large task.
Nathan Froyd had a productive couple of weeks.
- He added a memory reporter nsIDocument’s property tables.
- He measured some more stuff under the “xpti-working-set” reporter.
- He measured some more stuff under the “atom-table” reporter, renaming it “atom-tables” in the process.
- He measured some more stuff in the DOM multi-reporter.
These four reporters don’t cover a great deal of memory, but that’s inevitable these days; all the low-handing “heap-unclassified” fruit is gone, and the remaining improvements will necessarily be small. We’re gradually closing in on the target of 10%, though I’m not sure we’ll ever get there.
Nils Maier added some more identifying information to JavaScript sandboxes, which will help identify them in about:memory.
Bas Schouten added a memory reporter for VRAM used by Azure.
Fixed Leaks
Robert O’Callahan fixed a leak involving canvas, which I won’t pretend to understand. This also fixed bug 757749.
Cervantes Yu fixed a B2G memory leak that would leak about 1MB of memory per day. This was found by Valgrind, under the operation of Julian Seward.
Quote of the Week
A Hacker News thread contained the following comment.
Mozilla needs to focus on making its browser consume less memory and CPU when idle. These are the things that are killing its marketshare and leading users to switch to alternatives. Every time I see initiatives like the password account manager, iOS apps that use web view, a browser based editor and what not I just scratch my head because Firefox marketshare is plummeting and the old refrain “just wait until the next version” was tired since 3.0. Instead the general response is to get defensive and list new features. Yeah and are those new features stopping the user drain? Nope. Firefox is a sluggish memory sucking heap of software that drains resources even when it isn’t being used.
I love and cherish Firefox as a product and a brand, because you have achieved a lot in your past. But now you are failing and it is depressing to see this happen.
That’s not the quote of the week, BTW; I’ve heard variations of that many times before, and while it had some truth to it 18 months ago, that piece of conventional wisdom is now well past its use-by date. Pleasingly, there was quite a bit of push-back in response. My favourite comment, which is the quote of the week, was the following.
Firefox is to Ford what Chrome is to Toyota. Even if Firefox is as reliable and efficient as Chrome these days, some people still see it as being the gas-guzzling unreliable Ford vehicle of yesteryear.
A bad reputation is a difficult thing to lose.
Bug Counts
Here are the current bug counts.
- P1: 23 (-2/+0)
- P2: 87 (-10/+9)
- P3: 106 (-3/+3)
- Unprioritized: 2 (-2/+2)
A net change of -2 P1 bugs, and -1 P2 bugs. Not bad!
34 replies on “MemShrink progress, week 53-54”
The reputation won’t get better until those Firefox memory consumption bashing die off. In order to stop all the whinings, Moz should make clear it’s not Firefox what drains your memory, but your crappy add-ons you installed. Well, shouldn’t Moz provide some warnings or ratings of the hosted add-ons as a part of reviewing process? Individual add-on memory usage as part of about:memory is definitely a must.
The problem again, is with snappy. Until Mozilla Get SuperSnappy done and landed, and all other Snappy Bugs on High Priority, its reputation wont improve. Because Chrome has set a new standard in Browser Responsiveness. And Firefox, 3 years after Chrome initial release hasn’t managed to close the gap much.
Sad but true.
This soo true. But that is not the only thing that makes it hard to fight for Firefox. There are plenty of ideas, mockups and bugs filed, that get literal no high priority. Speed is not only the time it takes to render a certain webpage, but development too.
For example: Apple releases a new OS X version (10.7) with a lot new features (overflow scrollbars, new design, high DPI mode, double tag to zoom, auto resume, etc.) and 1 year later nothing has been implemented, but some half finished design pieces.
There bugs filed for unifying the experience that get no priority (e.g. finishing the Addon-Manger, finishing the Download Manager, integrating the search engine manager into Addon-Manager, etc.) and even design decisions made for version 4.0 are not implemented 1 year later.
Otherwise Mozilla announce things, but don’t release it. For example, pre-4.0 it was announced that Firefox 4.0 will be released as 64-bit for all platforms, but the truth is that Windows has no 64-bit version. Same goes with Electrolysis.
People remember this and still ask, when it will released. It seems that there is no focus anymore.
Additionally there were a ton of awful decisions: To switch to the rapid development model was great, but it took 1 year to finish the required work: make Updates silent (with staged updates).
Don’t get me wrong, I love Mozilla, Firefox and everything related, because of the mission and for the open web. I’m a proud supporter of Mozilla, but I think, that even hard-core fans are going to the alternatives, if there is not a true new focus.
I believe when Mozilla announce something, it should be implemented in no time. The Australis theme is one of those current projects: It get announced a while back, but the still platform bugs that needs to be fixed to finish the work (e.g. OS X title bar drawing)
This all seems off-topic, but I think it needs to be said. Chrome seems to be fast, because when there is new technology, it get implemented in no time.
Besides this fundamental project flaws, Firefox has still memory management flaws. The MemShrink Project did indeed had some breakthroughs like the better Garbage Collection timings, but that the memory consumptions are gaining with using the same profile for a while. When I start Firefox with a new Profile and all of my bookmarks and addons, it is indeed really very fast. But as time goes by, this profile makes Firefox slower and slower. Memory consumptions are gaining.
It got even worse with Firefox 13 and even more with Firefox 14. The best snappy / memory wise release was 12.
Anyway, You guys rock and thank you for the hard work!
Nick I’m not really sure what you hope to achieve by re-hashing the negative commentary of Firefox’s memory handling other than to perpetuate it? Without oxygen there is no smoke, let alone fire. Clearly the IT industry is full of uber nerds with troll-like diplomacy. It’s too easy to take sides, attack or defend. Both sides need to rise above that mentality (yes, this is moi talking, LOL) and stop the circle of bait and defend, then counterattack. I think it’s time that Mozilla started looking at honest and creative ways to overcome the genuine impressions of Firefox’s memory-eating, performance-destroying past. I’ve said it once already: be honest! Stop getting so defensive. Pretty much every non-generalized criticism of Firefox’s memory management has been proven correct by MemShrink so it doesn’t seem like Mozilla has any choice but to cough up and ask for the likely forgiveness that will arise. Mozilla really must come out and say something as blatantly honest as I might! 🙂 For example:
Surely something like that would be more effective than repeatedly quoting people who were actually right (otherwise there’d have been nothing for MemShrink to achieve) though perhaps not very diplomatic. The way I see it, generalizations about Firefox’s memory management in the past were akin to comments made by bitter and twisted couples in a divorce court. For many people, Firefox was their first love when it comes to browser software. When it seemed like Firefox was letting itself go and Mozilla – in this case playing the role of the defensive and obstructive in-laws who were in denial about their offspring’s part in the problem – was just throwing platitudes and excuses back at a once-loved partner, it was just too painful and bound to stir up all sorts of heated exchanges.
Mozilla needs to work hard and creatively to address the repairing of formerly once-in-a-lifetime loving relationships with it’s dedicated but now disillusioned users.
I’ll be clear from the start, I have no detailed knowlege about how much memory is ‘consumed’ by caching content in memory (beyond the known prefs) or how much memory, disk and time is consumed by the snap-shot ‘feature’ (thanks for the pref: browser.pagethumbnails.capturing_disabled to stop that, as there is no other way from stopping thumbnails to fill-up your disk-space), but that is room for improvement that is not especially tied to js, that is more tied to the ‘design’ and the ‘new’ ‘features’.
As said before in the comments: there is much old work still to be done, and nobody (of the visible devs) is pushing for completion.
E.g. the halfway done and then stopped ‘new’ download ‘library’. As it is, it should newer ever made into beta. Critical info is missing or can’t be shown because the tree-views are just copied from bookmark/history and nobody thought about extending them to the point of feature parity with the ‘old’ dedicated view( dl-start, dl-end, storage-location, dl-url, dl-start-page, etcetera).
Sorry for the rant, but some things get me angry beyond just frustrated.
Thanks for the immense work you and your team does.
Strongly agree with your download manager comment. I would love to get the old downloadmanager back in the nightlies because the new one is useless for anything I want to do. I didn’t even know it got to beta that is really stupid.
about:config -> browser.download.useToolkitUI -> true
Whats up with the post FF16 increase in memory consumption on areweslimyet.com?
I fail at threading, see my reply below.
“CPG will allow lots of things within Firefox to become simpler and faster. The main disadvantage is, unfortunately, increased memory consumption, as can be seen on areweslimyet.com.”
http://blog.mozilla.org/nnethercote/2012/05/16/memshrink-progress-week-47-48/
Yes I remember that, but I meant the newest increase (beginning of FireFox 16) which seems a little bit late to be fallout from CPG
We’re not sure. We’ve filed https://bugzilla.mozilla.org/show_bug.cgi?id=768716 for investigating it.
Nick, a request. Please consider setting the bugzilla links in your posts and comments to open in a new tab.
Thanks,
Manoj
I think it’s important to note that the memory consumption of Firefox has been pretty good since Firefox 7. Since then it has steadily been getting better and better.
The thing that still lets Firefox down is the poor memory consumption brought about by memory leaks from add-ons, but because these add-ons are part of Firefox, it’s the browser that gets the blame.
This however will cease to be an issue once Firefox 15 hits the stable release channel on or shortly after August 28th 2012, because of the legendary Hueyfix (bug 695480).
So once Firefox 15 is released add-on related memory issues will be removed from the equation, and anyone moaning about the memory usage will almost definitely be clinging to the past.
I use Firefox 13 or higher on 5 different computers (various home and work), all with no add-ons and it’s really fast and memory efficient.
Project Snappy has been in progress since November 2011 and the resultant work has landed in Firefox from version 11 onward. It has made a difference and will continue to make a difference.
Compartment-per-global landed in FF15 (the axis labelling is a bit ambiguous), I think that’s it. Our gracious host mentioned the increase here.
The comment about prioritization and focus is key. The fact that Mozilla does everything in the open has a big part to play in some of the frustration expressed in the comments. Mozilla needs to announce a few key Firefox initiatives a quarter, and deliver on them within the promised time-frame. The constant shifting of priorities creates unnecessary jank.
Users recognize that there are a number of things going on, and will not switch to an alternative as long as incremental value is demonstrated with each release. Unfortunately, this is not the case today. Not to beat on the DM re-design dead horse, but Limi blogged about the new DM design in April of 2011. This was a Firefox original idea; Safari has had a working implementation since June, 2011. Whither FF’s new, fully-functional DM? It is almost a year and we have a crippled DM in the nightlies that is styled poorly. Ditto for the new tab page; FF can and should borrow a page from Safari’s quasi-3D new tab page. That looks like it has been styled by someone with design chops.
Even though MemShrink is delivering constantly, end-users don’t really experience the full force of the changes made under-the-hood. Without Project Snappy becoming snappier with the rate of its fixes, the current perception of slowness is going to be hard to shake.
Wow, I’ve clearly hit a nerve this week.
With respect to the prioritization comments, all I can say is this: this is the way Mozilla works. Almost everything is done out in the open. That means that projects and features get described in advance. Sometimes they are delayed; sometimes they never finish. You’ll never have this problem with Safari, because Apple doesn’t tell you anything in advance about what will be in new releases.
As for all the dissatisfaction with non-MemShrink-related features, like the new download manager — this isn’t the place to complain about that. I have nothing to do with that stuff.
I have described those problem in comment exactly 1 year ago when MemShrink start. There is a difference between developing in the open, and announcing it to the public.
You See, Linux Kernel is developed in the open as well, yet Media dont pick up all the latest and greatest ideas on the spot and spread it like wildfire.
Let’s just hope Project Snappy will finally improve the perception of Firefox. Dont get me wrong, Memshrink did a GREAT Job.
One major difference is that with the kernel, features are iterated by contributors with diverse employers and generally with no timeframe or expectation that they will be pulled (merged) by someone closer to the mainline. That explains why they aren’t reported the same way as when an alpha feature lands in Firefox.
When CPG landed it didn’t seem to have any significant impact on memory use for me. Maybe it went up a little, but everything was pretty stable (and the ‘stable’ part was what was important). In the last couple weeks, however, memory usage (as reported by MemChaser) has been inflating badly. I’m at 700-800 MB just a couple hours after restarting (and most tabs from the previous session in the non-loaded state), whereas before it was stable at around 400-500 MB after a full day of use. (using FF16 nightly, btw)
Further, MemChaser’s memory report seems to often be cycling through a constant increase, from some baseline, then up 10-12 MB per tick (~2 seconds?), then dropping back down after an increase of about 100 MB, except that what it drops down to is a couple MB higher than what it started at. So it looks like a perpetual increase of a few MB per minute.
Looks like it might -possibly- be related to one of a couple of sites that I visit (though there’s no reason they should be causing this either), but I’d have to observe it some more to be sure. It’s not addons (or if it is, it’s a new interaction with a previously stable addon), since I’ve disabled all the addons I’ve added in the vaguely recent past.
And yes, Nic, a bit of a nerve there. Your little corner of Firefox makes steady progress (and -reports- steady progress) on a narrowly focused, important segment of the codebase. You identified (and perhaps even more important, acknowledged the existence of) a specific problem, and you’re solving that problem. The javascript team is likewise pretty well thought of. Some of the other segments, not so much, and as long as that negative perception persists for more visible projects, it’s going to continue to bleed over into the projects that may actually be improving.
Plus, internet time moves pretty slow when trying to improve people’s opinions, but incredibly fast when people’s opinions go down.
Nic, as David so eloquently put it, the Memshrink team has put focused effort into solving a major complaint with Firefox. The team did it with élan, and it has succeeded in turning Firefox’s reputation around.
The other teams – eh, not so much. And you are partially right; shifting priorities and unfinished projects are part of developing in the open. My observation is that this is more of a systemic problem at Mozilla – Thunderbird has its fair share of abandoned initiatives. You also said that this is how Mozilla works. It does not mean it has to continue this way. Look at the impact and progress Memshrink has made. There is a way to make tangible progress if there is the right mix of leadership and focus.
I might be speaking for the broader audience here, but the rest of the teams will be better served if they adopt the Memshrink-Way (TM). Set goals for themselves, and track their progress. Do it all in the open. Do it behind closed doors. But, hat tip to Nike, Just Do It. Already!
Hi Manoj,
If you would like to perhaps complete these abandoned projects, go right ahead.
Otherwise, realise that Mozilla are doing a bloody good job keeping up with web browsers built by a tonne people who are paid to do it by some of the biggest companies in the world, who have legions of paid beta testers and support.
Considering that many people who work on Mozilla and report bugs are doing it unpaid, in their spare time, and if somebody leaves because of life/lack of interest/etc that its hard to find somebody willing or able to continue the work.
Its been said before that closed source companies have incomplete and failed projects, you just don’t hear about them.
Go Mozilla!
The past reputation of Firefox is a ongoing problem with tech journalist who seems trained to parrot the same complaints. In their defense though, Firefox 4 was pretty bad, but it is inexcusable for them to spout the same stuff nowadays. It is worse b/c they always compare it to Chrome, but if you do a google search of “Chrome out of Memory”, Chrome has a frequent memory problem too.
If they actually used Firefox for any length of time, they would know better, but tech journalist have the worse journalistic standards next to the the paparazzi.
Hello, I’m a regular user of Firefox, I know nothing of coding, I barely understand these posts, but I keep up to date with Planet Mozilla Projects, Snappy, and Memshrink, the last two being my favorites as it talks more about fixing things.
I use Firefox as most users do, and I’ve rarely encountered problems. I got the add-on to check for slow startups, but never went past 8 seconds. Even if sometimes there are some problems, I will never leave Mozilla, because it stands for everything I believe in, the open web, community work for the benefit of everyone, not for greed. It is amazing to see the collaboration of people who aren’t paid to do this work, and how they push the web forward, even through mistakes, and give it all back to users for free, expecting nothing from us, and even share their findings freely to other competing companies.
Projects like Snappy and Memshrink are amazing, they bring focus and detailed accounts of work done so we can better understand the time it takes to implement things. Perhaps there should be more small teams and blogs built in the same way for other portions of projects. Anyway, thank you for your work and your posts, I greatly appreciate it, even if I barely understand most of it 😛
Thank you! It’s nice to receive positive feedback that isn’t of the form “I think you’re doing great stuff; BTW, I’m having a problem with X…” 🙂
Since April, the areweslimyet.com indicators seem to be climbing. Could you explain that changes that caused this? Will they go back to the April level eventually?
(I’m sorry if I missed you mentioning the reason in any earlier posts—I couldn’t find it.)
This has been discussed in this comment thread; you’ll also find a link to a previous post.
http://www.tomshardware.com/reviews/windows-xp-web-browser-performance,3167.html
Sorry, wrong link. Here is the correct one:
http://www.tomshardware.com/reviews/windows-7-chrome-20-firefox-13-opera-12,3228.html
Thanks for the link, Jim!
Yeah if you frill down to the details on the memory tests in the latest web browser grand prix on Toms Hardware you can see that Firefox is easily the memory king with the lowest 40 tab memory usages and not fairing too badly on single tab usage or memory management either.
http://www.tomshardware.com/gallery/memoryusage3wbgp11,0101-343668-11054-0-0-0-jpg-.html
I have suffered repeated crashes of firefox. The memory usage for firefox.exe, viewed with Win Task Manager, is over 1.5 GB (yes GB) with two windows and 22 tabs open. 33 are open now to hunt down this forum and search for answers. No other applications running. This after just 5.5 hours use. And this is not the remnants of downloaded pages. I’m on a very slow dialup line running at 26.4kb/s. To download 1GB would take over eight days uninterrupted. Clearly FF is logging/recording/copying something to memory or otherwise occupying RAM.
If you want to find a memory hog/drain/leak this is clearly a good one. Like the great white whale it is really big. If you can find you own ass with both hands you should be able to find this.
Sys – Win7-64 extensions up to date. All addons are up to date. And very common ones at that. 4GB RAM. Firefox version 13.0.1 , latest.
This is a long standing problem. Have had to stop using FF for business because of this and use Chrome. This is basic functionality and stability stuff. If you can’t get this right you can’t be trusted for anything. No combination of smooth, fast, pretty or feature rich can make up for failure in this area.
The crash reporter has sent many reports and not so much as a peep in return. I get the feeling that only preferred users get listened to. I’m not really expecting anything back. Response has been nonexistent.
e-mail at ao@gru.net.
Thanks for the report. I just replied to you via email.
Thank-you Nicholas Nethercote.
Good to see that at least one person here is not too busy slapping themselves on the back and congratulating each other to hear that FF is still something less than perfect. I would settle for it being as robust and reliable as it was five years ago.