Category: Firefox

Categories
AdBlock Plus Firefox Google Chrome

Converting a Chrome user to Firefox: a follow-up

I wrote yesterday about how I converted a relative of mine from a Chrome user to a Firefox user.  The post was picked up by the tech press including Tom’s Hardware and Conceivably Tech.  (Tom’s said I “described why getting users back from Chrome may nearly be impossible”, which I think is a laughable exaggeration of what I wrote.)

The good news is that the two major stumbling blocks I faced during the conversion are well on the way to being addressed.

  • The situation with third-party add-ons will be greatly improved in Firefox 8, which is scheduled for release in just a few days (2011-11-08).  The first time it runs, Firefox 8 will present the user with a window that lists all the installed add-ons, distinguishing between add-ons the user installed explicitly and third-party add-ons that they didn’t install explicitly.  The user will be asked to confirm which add-ons they want, and the third-party add-ons will be disabled by default.  The bug for this feature is here, and there is a follow-up bug here.
  • An “import history from Chrome” feature is being worked on right now.  Here is the feature page, a tracking bug, and  two dependent bugs.  The aim was for it to make Firefox 10, which is scheduled for release on 2012-01-31, but it looks like it might not make that release.  Hopefully it will make Firefox 11, which is scheduled for release on 2012-03-13.

Another point raised by commenters was that Chrome has a version of AdBlock Plus.  However, the Chrome version still has major shortcomings compared to the Firefox version.  This page states “We are currently working on providing the same experience for Google Chrome as what you are used to from Firefox. Please keep in mind that we are not there yet and much work still needs to be done. There are also known Google Chrome bugs and limitations that need to be resolved.”  This page lists the major shortcomings.  Maybe those shortcomings will be overcome in the future, but until they are, it’s not a reasonable comparison.

In conclusion, the point of my post yesterday was not to say “OMG Firefox is crap the sky is falling in”, but rather “here’s what happened when I tried this”.  I knew that the two obstacles I listed were being worked on, though I didn’t know the details.  (Many thanks to the commenters who filled me in.)  The fact that they are being worked on and/or have been fixed is rather encouraging.  It’s also worth noting that Firefox’s new rapid release calendar like these make it into the hands of ordinary users only 2 to 3 months after they are implemented.  With the old release calendar, these two improvements wouldn’t have made it into a release until mid-2012 or later.

Categories
AdBlock Plus Firefox Google Chrome Personal

Converting a Chrome user to Firefox

[Update: before commenting, you should probably read this follow-up post that clarifies certain things about this post.]

On my recent vacation I was staying with a family member, let’s call her Karen (not her real name).  She was a Google Chrome user, and I managed to convert her to a satisfied Firefox user.  Here’s what I learnt along the way.

tl;dr:

Bad things about the experience:

  • The third-party add-ons situation on Windows is awful.
  • We need a “import history from Chrome” feature.

Good things about the experience:

  • Mozilla’s non-profit nature is compelling, if you know about it.
  • AdBlock Plus is great.

The Initial Situation

Karen is a moderately sophisticated computer user. She knows what a browser is, but didn’t know how many there were, who made them, or any notable differences between them.

Her machine that is probably 2 or 3 years old, and runs Windows Vista.  She had used IE in the past (not sure which version) but didn’t like it, switched to Chrome at some point — she didn’t remember how or why — and found it to be much better.  She was running Chrome 14.0.835.202 (no, that’s not an IP address!) which was the latest stable version.

She also had Firefox 3.6.17 installed, but judging from the profile she hadn’t used it much — there was very little history.  She had the following Firefox add-ons installed:

  • Java Console 6.0.20 and 6.0.27.
  • The .NET Framework Assistant.
  • Some media player thing.
  • Some Norton “safe search” toolbar, and Symantec IPS, whatever that is.

(What is the Java Console?  What is the .NET Framework Assistant?  As far as I can tell they are (a) very common and (b) useless.)

I told her that I worked on Firefox and suggested that she try it and she was open to the idea.  I talked about the differences between Firefox and Chrome and some of my work on Firefox.  The thing that caught her attention most was that Mozilla is a non-profit organization.  She hadn’t known this and it appealed to her greatly — she said that browser speed and the non-profit nature were the two most important things to her.  She was also somewhat interested when I said that Firefox had an ad-blocking add-on.  At the end of the conversation, she agreed to let me install Firefox and make it the default browser.

Installing Firefox

I removed the existing Firefox profile manually — I wasn’t sure if this was necessary, but I definitely wanted a fresh profile — and then uninstalled Firefox through the Control Panel.  I then installed Firefox 7.0.1.  (BTW, I stayed with Karen for two weeks at this point, and I had deliberately waited until Firefox 7 was out before doing this because I knew it had much lower memory usage than Firefox 6.)

An unexpected thing was that the Firefox installer asked me to close all the other running programs;  it explained that this would mean that I wouldn’t have to reboot.  I’m used to running Firefox on Mac and Linux so I’m not used to this, but I’m familiar enough with Windows that I wasn’t totally surprised.  Still, it was annoying;  Karen had MS Word and some other programs open and I had to go ask her if I needed to save anything before closing them.  I realize this is Windows’ fault, not Firefox’s, but it was an obstacle.

Starting Firefox

When I started Firefox it asked me if I wanted to make it the default browser and I said yes.  (I explained to Karen how to switch the default browser back to Chrome if she was unhappy with Firefox.)

It also asked me if I wanted to import history/bookmarks/etc from IE.  But there was no equivalent for Chrome!  Karen had a ton of bookmarks in Chrome, but fortunately she said she only used a handful of them so I was able to copy them manually into Firefox’s bookmarks toolbar (which I had to make visible).  I’ve heard that someone is working on an “import from Chrome”  feature to Firefox but I don’t know what the status is.  We need it badly.

Once Firefox started, another unexpected thing was the state of the add-ons in the new profile.  The Symantec add-ons (including the ugly Norton toolbar) were present and enabled.  I had to disable them in about:addons;  I wanted to completely uninstall them but I couldn’t, the “uninstall” button just wasn’t present.  The Java Consoles and the media player were disabled because they were incompatible with 7.0.1, but I was also not able to  uninstall them.  This horrified me.  Is it a Windows-only behaviour?  Whatever the explanation, the default situation in a fresh install was that Firefox had several unnecessary, ugly additions, and it took some effort to remove them.  I’m really hoping that the add-on confirmation screen that has been added to Firefox 8 will help improve this situation, because this was the single worst part of the process.

I then tweaked the location of the home and reload buttons so they were in exactly the same position as in Chrome.  I probably didn’t need to do that, but  with those changes made Firefox’s UI looked very similar to Chrome’s, and I wanted things to be as comfortable for her as possible.

The best part of the process was when I installed AdBlock Plus.  With Karen watching, I visited nytimes.com in Chrome, and I had to skip past a video advertisement before even getting to the front page, and then the front page had heaps of ads.  Then I visited in Firefox — no video, no ads.  It was great!

Follow-up

A week or two later I sent Karen a follow-up email to check that everything was ok.  She said “All is well! The ad blocking is a great feature.”

So, Firefox has a new and happy user, but there were some obstacles along the way, and the outcome probably wouldn’t have been so good if Karen hadn’t had a Firefox expert to help her.

Categories
Firefox JägerMonkey Memory consumption MemShrink Tracemonkey Uncategorized

MemShrink progress, week 20

Surprise of the week

[Update: This analysis about livemarks may be wrong.  Talos results from early September show that the MaxHeaps number increased, and the reduction when the “Latest Headlines” livemark was removed has undone that increase.  So livemarks may not be responsible at all, it could just be a coincidence.  More investigation is needed!]

Jeff Muizelaar removed the “Latest Headlines” live bookmark from new profiles.  This was in the Bookmarks Toolbar, and so hasn’t been visible since Firefox 4.0, and most people don’t use it.  And it uses a non-zero amount of memory and CPU.  Just how non-zero was unclear until Marco Bonardo noticed some big performance improvements.  First, in the Talos “MaxHeap” results on WinNT5.2:

Talos MaxHeap graph

And secondly in the Talos “Allocs” results on WinNT5.2 and Mac10.5.2:

Talos Allocs graph

In the WinNT5.2 cases, it looks like we had a bi-modal distribution previously, and this patch changed things so that the higher of the two cases never occurred.  In the Mac10.5.2 case we just had a simple reduction in the number of allocations.  On Linux the results were less conclusive, but there may have been a similar if smaller effect.

This surprised me greatly.  I’ve done a lot of memory profiling of Firefox and never seen anything that pointed to the feed reader as using a lot of memory.  This may be because the feed reader’s usage is falling into a larger, innocuous bucket, such as JS or generic strings.  Or maybe I just missed the signs altogether.

Some conclusions and questions:

  • If you have live bookmarks in your bookmarks toolbar, remove them! [Update: I meant to say “unused live bookmarks”.]
  • We need to work out what is going on with the feed reader, and optimize its memory usage.
  • Can we disable unused live bookmarks for existing users?

Apparently nobody really owns the feed reader, because previous contributors to it have all moved on.  So I’m planning to investigate, but I don’t know the first thing about it.  Any help would be welcome!

 Other stuff

There was a huge memory leak in the Video DownloadHelper add-on v4.9.5, and possibly earlier versions.  This has been fixed in v4.9.6a3 and the fix will make it into the final version v4.9.6 when it is released.  That’s one more add-on leak down, I wonder how many more there are to go.

TraceMonkey, the trace JIT, is no longer built by default.  This means it’s no longer used, and this saves both code and data space.  The old combination of TraceMonkey and JaegerMonkey is slower than the new combination of JaegerMonkey with type inference, and TraceMonkey is also preventing various clean-ups and simplifications, so it’ll be removed entirely soon.

I refined the JS memory reporters in about:memory to give more information about objects, shapes and property tables.

I avoided creating small property tables, removed KidHashes when possible, and reduced the size of KidHashes with few entries.

I wrote about various upcoming memory optimizations in the JS engine.

Justin Lebar enabled jemalloc on MacOS 10.5 builds.  This was expected to be a space improvement, but it also reduced the “Tp5 MozAfterPaint” page loading benchmark by 8%.

Robert O’Callahan avoided excessive memory usage in certain DOM animations on Windows.

Drew Willcoxon avoided excessive memory usage in context menu items created by the add-on SDK.

Bug Counts

  • P1: 35 (-1, +1)
  • P2: 116 (-2, +5)
  • P3: 55 (-2, +3)
  • Unprioritized: 5 (-4, +5)

At this week’s MemShrink meeting we only had 9 bugs to triage, which is the lowest we’ve had in a long time.  It feels like the MemShrink bug list is growing slower than in the past.

Categories
about:memory DMD Firefox Valgrind

Reducing about:memory’s “heap-unclassified” measurement with DMD

about:memory is a really useful tool, but everyone always complains about the “heap-unclassified” number being too high.  It’s too high because we don’t have enough memory reporters implemented.

DMD is a tool that identifies where new memory reporters should be added to reduce “heap-unclassified”.  I’ve been using it for a couple of months now, and I’ve just written instructions on how to use it.  It’s not easy, but hopefully it’s doable.

It’s probably worth pointing out that we have a pretty good handle on what needs to be done to reduce “heap-unclassified” significantly — check out the list of memory reporters to be implemented.  Almost all of those bugs were filed based on data from DMD.  The single most important bug in that list is the one to add missing style reporters;  I see multiple megabytes of CSS stuff all the time in DMD’s output.

Categories
about:memory Firefox Garbage Collection JägerMonkey Memory consumption MemShrink Tracemonkey

SpiderMonkey is on a diet

One thing I’ve learnt while working for Mozilla is that a web browser can be characterized as a JavaScript execution environment that happens to have some multimedia capabilities.  In particular, if you look at Firefox’s about:memory page, the JS engine is very often the component responsible for consuming the most memory.

Consider the following snapshot from about:memory of the memory used by a single JavaScript compartment.

about:memory snapshot

(For those of you who have looked at about:memory before, some of those entries may look unfamiliar, because I landed a patch to refine the JS memory reporters late last week.)

There is work underway to reduce many of the entries in that snapshot.  SpiderMonkey is on a diet.

Objects

Objects are the primary data structure used in JS programs;  after all, it is an object-oriented language.  Inside SpiderMonkey, each object is represented by a JSObject, which holds basic information, and possibly a slots array, which holds the object’s properties. The memory consumption for all JSObjects is measured by the “gc-heap/objects/non-function” and “gc-heap/objects/function” entries in about:memory, and the slots arrays are measured by the “object-slots” entries.

The size of a non-function JSObject is currently 40 bytes on 32-bit platforms and 72 bytes on 64-bit platforms.  Brian Hackett is working to reduce that to 16 bytes and 32 bytes respectively. Function JSObjects are a little larger, being (internally) a sub-class of JSObject called JSFunction.  JSFunctions will therefore benefit from the shrinking of JSObject, and Brian is slimming down the function-specific parts as well.  In fact, these changes are complete in the JaegerMonkey repository, and will likely be merged into mozilla-central early in the Firefox 11 development period.

As for the slots arrays, they are currently arrays of “fatvals” A fatval is a 64-bit internal representation that can hold any JS value — number, object, string, whatever.  (See here for details, scroll down to “Mozilla’s New JavaScript Value Representation”;  the original blog entry is apparently no longer available).  64-bits per entry is overkill if you know, for example, that you have an array full entirely of integers that could fit into 32 bits.  Luke Wagner and Brian Hackett have been discussing a specialized representation to take advantage of such cases.  Variations on this idea have been tried twice before and failed, but perhaps SpiderMonkey’s new type inference support will provide the right infrastructure for it to happen.

Shapes

There are a number of data structures within SpiderMonkey dedicated to making object property accesses fast.  The most important of these are Shapes.  Each Shape corresponds to a particular property that is present in one or more JS objects.  Furthermore, Shapes are linked into linear sequences called “shape lineages”, which describe object layouts.  Some shape lineages are shared and live in “property trees”.  Other shape lineages are unshared and belong to a single JS object;  these are “in dictionary mode”.

The “shapes/tree” and “shapes/dict” entries in about:memory measure the memory consumption for all Shapes.  Shapes of both kinds are the same size;  currently they are 40 bytes on 32-bit platforms and 64 bytes on 64-bit platforms.  But Brian Hackett has also been taking a hatchet to Shape, reducing them to 24 bytes and 40 bytes respectively.  This has required the creation of a new auxiliary BaseShape type, but there should be many fewer BaseShapes than there are Shapes.  This change will also increase the number of Shapes, but should result in a space saving overall.

SpiderMonkey often has to search shape lineages, and for lineages that are hot it creates an auxiliary hash table, called a “property table”, that makes lookups faster.  The “shapes-extra/tree-tables” and “shapes-extra/dict-tables” entries in about:memory measure these tables.  Last Friday I landed a patch that avoids building these tables if they only have a few items in them;  in that case a linear search is just as good.  This reduced the amount of memory consumed by property tables by about 20%.

I mentioned that many Shapes are in property trees.  These are N-ary trees, but most Shapes in them have zero or one child;  only a small fraction have more than that, but the maximum N can be hundreds or even thousands.  So there’s a long-standing space optimization where each shape contains (via a union) a single Shape pointer which is used if it has zero or one child.  But if the number of children increases to 2 or more, this is changed into a pointer to a hash table, which contains pointers to the N children.  Until recently, if a Shape had a child deleted and that reduced the number of children from 2 to 1, it wouldn’t be converted from the hash form back to the single-pointer.  I changed this last Friday.  I also reduced the minimum size of these hash tables from 16 to 4, which saves a lot of space because most of them only have 2 or 3 entries.  These two changes together reduced the size of the “shapes-extra/tree-shape-kids” entry in about:memory by roughly 30–50%.

Scripts

Internally, a JSScript represents (more or less) the code of a JS function, including things like the internal bytecode that SpiderMonkey generates for it.  The memory used by JSScripts is measured by the “gc-heap/scripts” and “script-data” entries in about:memory.

Luke Wagner did some measurements recently that showed that most (70–80%) JSScripts created in the browser are never run.  In hindsight, this isn’t so surprising — many websites load libraries like jQuery but only use a fraction of the functions in those libraries.  It wouldn’t be easy, but if SpiderMonkey could be changed to generate bytecode for scripts lazily, it could reduce “script-data” memory usage by 60–70%, as well as shaving non-trivial amounts of time when rendering pages.

Trace JIT

TraceMonkey is SpiderMonkey’s original JIT compiler, which was introduced in Firefox 3.5.  Its memory consumption is measured by the “tjit-*” entries in about:memory.

With the improvements that type inference made to JaegerMonkey, TraceMonkey simply isn’t needed any more.  Furthermore, it’s a big hairball that few if any JS team members will be sad to say goodbye to.  (js/src/jstracer.cpp alone is over 17,000 lines and over half a megabyte of code!)

TraceMonkey was turned off for web content JS code when type inference landed.  And then it was turned off for chrome code.  And now it is not even built by default.  (The about:memory snapshot above was from a build just before it was turned off.)  And it will be removed entirely early in the Firefox 11 development period.

As well as saving memory for trace JIT code and data (including the wasteful ballast hack required to avoid OOM crashes in Nanojit, ugh), removing all that code will significantly shrink the size of Firefox’s code.  David Anderson told me the binary of the standalone JS shell is about 0.5MB smaller with the trace JIT removed.

Method JIT

JaegerMonkey is SpiderMonkey’s second JIT compiler, which was introduced in Firefox 4.0.  Its memory consumption is measured by the “mjit-code/*” and “mjit-data” entries in about:memory.

JaegerMonkey generates a lot of code.  This situation will hopefully improve with the introduction of IonMonkey, which is SpiderMonkey’s third JIT compiler.  IonMonkey is still in early development and won’t be integrated for some time, but it should generate code that is not only much faster, but much smaller.

GC HEAP

There is a great deal of work being done on the JS garbage collector, by Bill McCloskey, Chris Leary, Terrence Cole, and others.  I’ll just point out two long-term goals that should reduce memory consumption significantly.

First, the JS heap currently has a great deal of wasted space due to fragmentation, i.e. intermingling of used and unused memory.  Once moving GC — i.e. the ability to move things on the heap — is implemented, it will pave the way for a compacting GC, which is one that can move live things that are intermingled with unused memory into contiguous chunks of memory.  This is a challenging goal, especially given Firefox’s high level of interaction between JS and C++ code (because moving C++ objects is not feasible), but one that could result in very large savings, greatly reducing the “gc-heap/arena/unused” and “gc-heap-chunk-*-unused” measurements in about:memory.

Second, a moving GC is a prerequisite for a generational GC, which allocates new things in a small chunk of memory called a “nursery”.  The nursery is garbage-collected frequently (this is cheap because it’s small), and objects in the nursery that survive a collection are promoted to a “tenured generation”.  Generational GC is a win because in practice the majority of things allocated die quickly and are not promoted to the tenured generation.  This means the heap will grow more slowly.

Is that all?

It’s all I can think of right now.  If I’ve missed anything, please add details in the comments.

There’s an incredible amount of work being done on SpiderMonkey at the moment, and a lot of it will help reduce Firefox’s memory consumption.  I can’t wait to see what SpiderMonkey looks like in 6 months!

Categories
Fennec Firefox Garbage Collection Memory consumption MemShrink

MemShrink progress, week 19

Good News

Peter Van der Beken fixed a leak caused by chrome code that injects a function into pages.  This was a MemShrink P1 bug.  The commentary in the bug is confusing, but this may have been affecting numerous add-ons including Firebug. (Bug 669730 is open for tracking leaks in Firebug;  it hasn’t yet been confirmed whether this fix has helped Firebug.)

Andrew McCreight rewrote JS_DumpHeap so that it dumps the complete JS object graph.  This was a MemShrink P1 bug because it’s an important piece of infrastructure for writing leak detection tools.

Justin Lebar fixed the measurement of RSS on Mac.  This was a MemShrink P1 bug because it prevented us from enabling jemalloc on Mac 10.5 machines.

Brian Hackett tweaked the JS engine so that more method JIT code can be discarded periodically, particularly chrome code in system compartments.

Oleg Romashin avoided using transparent layers in Fennec remote offscreen viewports.  This saves 5MB of memory in the active tab.

I added a memory reporter for the startup cache.  This is often around 1MB of memory, but it’s allocated via mmap/VirtualAlloc and so doesn’t change the “heap-unclassified” number in about:memory.

Boris Zbarsky fixed a small leak involving CSS transforms.

Bad News

It appears a bad memory regression has occurred in the past week or so.  Several people have reported multi-second pauses caused by garbage collection and cycle collection.  The problem only seems to strike when many tabs are open and/or the browser has been running for multiple days.  This needs investigation;  if you are experiencing similar problems please report in the bug.  Reliable steps to reproduce this bug will be invaluable.  If you turn on javascript.options.mem.log in about:config you can see when GCs and CCs occur and how long they take, which is helpful for diagnosis.

Bug Counts

The current bug counts are as follows.

  • P1: 35 (-5, +3)
  • P2: 113 (-3, +8)
  • P3: 54 (-0, +1)
  • Unprioritized: 4 (-5, +3)

I want to point out this bug, which presents an idea to help hunt down reproducible leaks that occur when users have multiple add-ons present.  This is important because many of the leaks reported by users recently are due to add-ons, but often the reporter has many add-ons installed which makes finding the culprit painful.  The goal is to write a Firefox add-on that selectively disables installed add-ons, so that a user can bisect them to discover which add-on is responsible for the leak.  This bisecting process is something that people can do manually, but an add-on that automates the process would make things easier and less error-prone.  Mercurial’s ‘hg bisect’ command would serve as a useful comparison.  Ehsan Akhgari has volunteered to mentor anyone who would like to try to implement this.

 

Categories
Fennec Firefox Memory consumption MemShrink

MemShrink Progress, weeks 13–18

I’ve been on vacation, so this report covers (incompletely) the past six weeks’ worth of MemShrink improvements.

Big Things

Paul Biggar and Mike Hommey enabled jemalloc on Mac 10.6.  This will hopefully reduce fragmentation on that platform.  And it brings it in line with Windows, Linux and Android.  Now we just need it for Mac 10.5 and Mac 10.7.

Oleg Romashin found a way to drop some Thebes layers in inactive tabs in Fennec.  I won’t pretend to understand the details of this bug, but if I’ve understood correctly it can saves 12MB or more per background tab.

Jeff Muizelaar turned on decode-on-draw.  This means that if you open a new page in a background tab, none of its images will be decoded until you switch to that tab.  Previously any images would be decoded and then discarded after 10 to 20 seconds (unless you switched to the tab before the time-out occurred).  This change can save a lot of memory (and CPU time) for users browsing image-heavy sites.

Gian-Carlo Pascutto optimized the safe browsing database.  This hopefully has fixed our url-classifier bloat problems.  (I plan to verify this soon.)

Chris Leary and Jonathan “Duke” Leto made regexp compilation lazy.  This potentially saves 10s or even 100s of MBs of memory in some cases by not compiling some regexps, and also allowing regexps to be GC’d more quickly.  There were some possible performance regressions from this patch, it’s unclear from the bug what exactly the state of these are.

Justin Lebar converted some uses of nsTArray to nsAutoTArray (and also here).  These avoided some calls to malloc (in the latter case, around 3% of all malloc calls!) and may help reduce heap fragmentation a little.  Robert O’Callahan did a separate, similar nsAutoTArray change here.  Justin also avoided another 1% of total malloc calls in JSAutoEnterCompartment.

Chris Leary rewrote JSArena, which avoided some wasted memory, as well as replacing some hoary old C code with sleek modern C++.

SMALLER THINGS

A new contributor named Jiten (a.k.a. deLta30) fixed about:memory’s GC and CC buttons so they trigger collections in both the content and chrome process in Fennec.  Previously only the chrome process was affected.  (I’m not sure how this change will be affected by the decision to switch to the native chrome process in Fennec.)  Great work, Jiten!

I avoided some wasted space in the JS code generator, and some more in the parser.  Justin Lebar did something similar in nsTArray_base.

Jonathan Kew added a memory reporter for textruns and associated data.  Justin Lebar added the “history-links-hashtable” memory reporter.

Justin Lebar fixed some bogus “canvas-2d-pixel-bytes” values in about:memory.

Brian Bondy fixed a leak in Windows drag and drop code.

Tim Taubert wrote about finding leaks in browser-chrome mochitests.

Bug Counts

The current bug counts are as follows.  The differences are against the MemShrink week 12 counts.

  • P1: 37 (-3, +11)
  • P2: 108 (-9, +37)
  • P3: 53 (-1, +14)
  • Unprioritized: 6 (-21, +5)

They’re still going up.  The good news is that my gut feeling is that not many of these bugs are problems reported by users.  (And those that are often are caused by add-ons.)  Most of the new reports are ideas for improvements from developers.

Categories
about:memory Firefox Memory consumption MemShrink

MemShrink progress, week 12

about:memory improvements

Lots of changes were made to about:memory this week.

Justin Lebar landed a patch that provides detailed information about RSS, vsize and swap memory usage on Linux and Android.  (The patch was backed out due to a minor leak but I expect Justin will fix that and re-land it soon.)  This will help us understand memory usage that is not covered by the “Explicit Allocations” tree in about:memory, such as memory used for static code and data, and it should be particularly useful on Android.  The contents of the new trees are hidden by default;  you have to click on the tree heading to expand each one.

Kyle Huey split up about:memory’s layout measurements on a per-PresShell basis.  This makes it easy to see how much layout memory is being used by each web page.

Something I failed to mention last week was that with the landing of type inference, each JavaScript compartment has five new measurements: “object-main”, “script-main”, “tables”, “analysis-temporary”, and “object-empty-shapes”.

I converted some of the JavaScript memory reporters to use moz_malloc_usable_size to measure actual allocation sizes instead of requested allocation sizes.  This accounts for slop bytes caused by the heap allocator rounding up.  This is quite important — slop bytes can easily account for over 10% of the heap, and if we don’t account for them we’ll never get about:memory’s “heap-unclassified” number down.  Therefore I’ll be doing more of this in the future.  And it would be great if people writing new memory reporters can do the same thing!

Finally, on the topic of “heap-unclassified” number:  people often complain about it, so I’m happy to say that it is on a clear downward path.  Indeed, thanks to DMD, at the time of writing we have 16 open bugs to add new memory reporters for things that consume significant amounts of memory, and 13 of these are assigned.  I’m hoping that in a month or two the “heap-unclassified” number on development builds will typically be 10–15% rather than the 30–35% it usually is now.

Other things

I changed the growth strategy used for one of JaegerMonkey’s buffers to avoid large amounts of memory wasted due to slop bytes.  These buffers are short-lived so the fix doesn’t make a great difference to total memory consumption, but it does reduce the number of allocations and heap churn.

Marco Bonardo wrote about his recent changes to the handling of the places database.

Dietrich Ayala wrote about an experimental add-on that unloads tabs that haven’t been viewed in a while, which is an interesting idea.  I suspect the exact approach used in the add-on won’t be viable in the long run, but we can certainly benefit from doing a better job of discarding regenerable info that hasn’t been used in a while, particularly on mobile.

Bug counts

This weeks’s bug counts are as follows:

  • P1: 29 (-2, +2)
  • P2: 80 (-4, +8)
  • P3: 40 (-2, +4)
  • Unprioritized: 22 (-12, +12)

Just like last week, Marco Castelluccio tagged quite a lot of old bugs with “[MemShrink]”.  We had 45 unprioritized bugs at the start of this week’s meeting, and we got through more than 20 of them.

Some comments on last week’s post got me thinking about how to make it easier for more people to help with MemShrink.  For those who don’t have much coding experience, probably the best bet is to look at the list of unconfirmed bugs — these are problems reported by users where the particular problem hasn’t been identified.  Often they need additional effort to determine if they are reproducible, due to add-ons, etc.  For example, in bug 676872 a user was seeing very high memory usage, and it’s clear that it was caused by one or more of the 41(!) add-ons he had enabled.  Ideally that bug’s reporter would disable them selectively to narrow that down, but anyone could do likewise with some effort.

For those who do have coding experience, it would be worth looking at the list of bugs that have a “mentor” annotation.  For example, bug 472209 is about adding some graphing capability to about:memory.  Jezreel Ng made some excellent progress on this during his internship, it just needs someone to take over and finish it up.

Finally, for those who like a challenge or have some experience with Firefox’s code, the full list of unassigned bugs might be of interest.  There are currently 86 such bugs!  More than I’d like.

(BTW, I’ve added links for the three bug lists above to the MemShrink wiki page.)

On HIATUS

I will be on vacation for the next five weeks and the MemShrink progress report will be on hiatus during that time.  But MemShrink meetings will continue (except there won’t be one next week due to the Mozilla all-hands meeting).  I look forward to writing a bumper progress report on October 19, where I’ll be able to summarize everything that happened while I was away!

Categories
about:memory Firefox JägerMonkey Memory consumption MemShrink SQLite

MemShrink progress, week 11

This week was quiet in terms of patches landed.

  • Marco Bonardo changed the way the places.sqlite database is handled. I’m reluctant to describe the change in much detail because I’ll probably get something wrong, and Marco told me he’s planning to write a blog post about it soon.  So I’ll just quote from the bug: “Globally on my system (8GBs) I’ve often seen places.sqlite cache going over 100MB, with the patch I plan to force a maximum of 60MB (remember this will vary based on hardware specs), that is a >40% improvement. We may further reduce in future but better being on the safe side for now.”  This was a MemShrink:P1 bug.
  • New contributor Sander van Veen knocked off another bug (with help from his friend Bas Weelinck) when he added more detail to the “mjit-code” entries in about:memory.  This makes it clear how much of JaegerMonkey’s code memory usage is for normal methods vs. memory for compiled regular expressions.
  • I rearranged nsCSSCompressedDataBlock to avoid some unnecessary padding on 64-bit platforms.  This can save a megabyte or two if you have several CSS-heavy (e.g. Gmail) tabs open.   It makes no difference on 32-bit platforms.

But it was a very busy week in terms of bug activity.  Let’s look at the numbers.

  • P1: 29 (-2, +2)
  • P2: 76 (-10, +20)
  • P3: 38 (-1, +2)
  • Unprioritized: 22 (-5, +23)

Several things happened here.

  • Marco Castelluccio looked through old bugs and found a lot (30 or more) that were related to memory usage and tagged them with “MemShrink”.
  • Nine new bugs were filed to reduce about:memory’s “heap-unclassified” number by adding memory reporters;  many of these were thanks to Boris Zbarsky’s insights into the output produced by DMD.
  • I closed out a number of bugs that were incomplete, stale, or finished;  this included some of those newly marked by Marco, and some ones that were already tagged with “MemShrink”.
  • I tagged five leaks that were found with the cppcheck static analysis tool.

We spent the entire MemShrink meeting today triaging unprioritized bugs and we got through 23 of them.  Of the remaining unprioritized bugs, the older ones tagged by Marco and the cppcheck ones (which I tagged after the meeting) constitute most of them.

It’s clear that the rate of problem/improvement identification is outstripping the rate of fixes.  We have a standing agenda item in MemShrink meetings to go through Steve Fink’s ideas list, but we haven’t touched it in the past two meetings because we’ve spent the entire time on triage.  And when we do go through that list, it will only result in more bugs being filed.  I’m hoping that this glut of MemShrink-tagged bugs is temporary and the new bug rate will slow again in the coming weeks.

In the meantime, if you want to help, please look through the lists of open bugs, or contact me if you aren’t sure where to start, and I’ll do my best to find something you can work on.  Thanks!

Categories
Firefox Memory consumption

“Browser X is using Y MB of memory with Z tabs open” is a meaningless observation

I read an exchange today about the memory usage of Firefox 7 (which is currently in beta).

It uses 600 MB of RAM with 10 tabs open (different websites).

lol wut? I’m not getting the same results as you. I have 12 tabs open here… RAM usage rarely exceeded 300mb for me. That’s already much better than Chrome, which seems to be using 190mb for just 2 tabs.

Memory consumption is around 300 MB for me too.

I’ve seen variants of this exchange countless times.  Unfortunately, observations such as “browser X is using Y MB of memory with Z tabs open” are meaningless.  To understand why, consider this alternative, hypothetical exchange:

I ran 3 different programs on my computer, they took 55 seconds to finish.

lol wut?  I just ran 8 programs and they only took 20 seconds to finish!

You’d never see an exchange like that, because it’s obvious that a comparison of the run-time of multiple programs is meaningless if you don’t specify what those programs are.  Workload matters. And yet, people make comparisons of browser memory usage like this all the time.

Here’s a simple example to show why this isn’t meaningful.  I just did some memory measurements with a development build of Firefox 9, using a moderately-used profile with no add-ons installed.  First I started it and opened 10 instances of www.google.com.au, one per tab, and the resident set size (RSS) was 139MB.  Then I re-started it and opened 10 instances of gmail.com, and the RSS climbed to 651MB.

This is hardly surprising.  gmail.com is a fully-fledged email client.  www.google.com.au is not much more than a search box.

Why do people so often make these meaningless “Y MB of memory with Z tabs open” comments?  My theory is that it’s because most people have no idea how complicated web browsers and web pages are.  Mild experience with HTML authoring is endemic — heaps of people have thrown together a basic web page, and it’s just a document with with some structure and styling, right?  So they think the difference between the website they wrote for their scout troop in 1997 (the one with the “under construction” animated GIF) and the front page of TechCrunch is merely a difference of degree, not kind.

They’re wrong.  For one, HTML and related technologies are hugely more powerful and complicated now than they were a few years ago.  Also, their scout troop website probably didn’t contain multiple megabytes of JavaScript code tracking its visitors’ every move. A web browser is not a document viewer, it is a full-blown programming environment with some very sophisticated text and graphical capabilities.  A web page is not a document but a program.  Therefore, the memory (and CPU) usage of different web pages varies dramatically.

Here’s a check-list of information that you should include if you want an observation about browser memory usage to be meaningful.

  • What sites do you have open?  The more specific, the better.  E.g. “Gmail and a few nytimes.com pages” is ok, but listing the exact URLs is better.
  • How did the browser get in this state?  Did you just start it, or have you been using it and visiting other sites for hours?
  • What memory metric are you using, and how did you measure it?  “Explicit” from Firefox’s about:memory?  “Private bytes” from the Windows Task Manager?  “RSS” from “top” on Mac or Linux?  (The full output of Firefox’s about:memory is hugely useful, because it includes dozens of highly relevant measurements.)
  • In Firefox, do you have any add-ons installed?  That can (and often does) make a big difference.  (And that’s a topic for another day.)

If you include all that, it’s highly likely that somebody else can reproduce your measurements, which means it’s a highly meaningful observation.

On a related note, I wrote previously about Firefox 7’s memory improvements, saying “Firefox 7 uses less memory than Firefox 6 (and 5 and 4): often 20% to 30% less, and sometimes as much as 50% less.”  The presence of the words “often” and “sometimes” in that sentence were deliberate.  We’ve seen those numbers in our testing, but once again, workload matters.  I really hope the numbers we’ve seen match what normal users see.  In fact, I thought that people would independently test these claims shortly after I wrote them.  But, to my knowledge, that hasn’t happened so far, even though those claimed improvements have been reported far and wide.  (I’ve even seen numerous headlines that say “Firefox 7 to use 50% less memory”, alas.)  Maybe once Firefox 7 is officially released people will make independent measurements.

(A final note:  suspicious readers may think that I’m trying to obliquely absolve Firefox from any responsibility for its memory usage by blaming web pages instead.  I’m not.  Firefox is doing much better now, but there’s still plenty of ways it can be improved;  please contact me if you want to help.)