I’ve joked before that I hoped to never work on anything in Firefox that requires me to build the browser regularly. That probably sounds weird unless you are a member of the JavaScript team, because it’s possible to do close to 100% of SpiderMonkey development just building the JS shell.
Working on the JS shell is great. On my 2.5 year old Linux box it takes maybe 1 minute to build from scratch, rebuilds are almost instantaneous, the regression tests take a few minutes, shell start-up is instantaneous, you rarely have to do try server runs, tools like GDB and Valgrind are easy to run, and landing patches on the TraceMonkey repo is low-stress because breakage doesn’t affect too many people.
In comparison, working on the browser sucks. Builds from scratch take 25 minutes, zero-change rebuilds take 1.5 minutes, single-change rebuilds take 3 or 4 minutes, the linking stage grinds my machine to a halt, cold starts take up to 20 seconds or more (warm starts are much better), the test suites are gargantuan, every change requires a try server run, tools like GDB and Valgrind require jumping though hoops (--disable-jemalloc, anyone?), and landing patches on mozilla-central is stressful.
Thanks to my recent about:memory work, I’ve had to experience this pain first-hand. It’s awful. Debugging experiments that would take 20 seconds in the shell take 5 minutes in the browser. I avoid ‘hg up’ as much as possible due to the slow rebuilds it usually entails. How do all you non-JS people deal with it? Maybe you just get used to it… but I figure there have to be some tips and tricks I’m not aware of.
(Nb: Why are rebuilds so slow? Because configure is invoked every time? Imprecise dependencies in makefiles? Too much use of recursive make? Bug 629668? Why do Fennec rebuilds seem to be slower than Firefox rebuilds?)
Kyle Huey told me how you can get away with rebuilding only parts of the browser. Eg. if I only modify code under xpcom/, I can just do make -C <build>/xpcom && make -C <build>/toolkit/library and this reduces the rebuild time a bit. The down-side is that when I screw it up, eg. by forgetting to rebuild a directoy that I changed, it creates Frankenbuilds, and realizing what I’ve done can end up taking a lot more time than I saved.
Another trick I worked out: implement things in JavaScript. Seriously! While doing my about:memory revamp I originally had a big chunk of the work done on the C++ side. I quickly realized that if I did most of it on the JavaScript side I could see the effect of most changes by simply copying aboutMemory.js from my source dir to my build dir and then reloading the page. Much better than re-building and re-starting.
What else can I do? Get a faster machine is the obvious option, I guess. More cores would help, though linking would still be a bottleneck. Do SSDs make a big difference?
Also, there’s talk of using more project branches and introducing mozilla-staging. That would avoid the stress of landing on mozilla-central, but that’s really the smallest part of what I’m complaining about.
Any and all suggestions are welcome! Please, I’m begging you.
34 replies on “Working on the browser sucks”
In my experience:
SSDs don’t help much. Moving from 2 to 4 cores helps, but beyond that, not so much. Faster (newer) CPUs help.
The Gold linker helps, although I have problems with it plus Valgrind. You need to have a Valgrind mozconfig anyway, so you can just disable Gold in there.
8GB RAM helps. And if you’re like me and qpush and qpop a lot, ccache helps.
Writing a script to do those incremental builds can prevent screwups.
Justin: thanks for the info. How do I enable gold? I had ccache on at one point but it didn’t help with my JS development, but maybe it’s worth re-enabling now. And I have scripts to do the incremental builds — the problem isn’t me making typos, it’s me forgetting that I modified directory Y as well as directory X 🙂
“the problem isn’t me making typos, it’s me forgetting that I modified directory Y as well as directory X”. Though it’s sad I have to remember that at all.
I work on multiple things at once. Either in multiple source trees, on different patches, or on Python scripts that analyze logs. The difference in the edit-compile-run cycle is pretty stark, there. It does make you think carefully about how you can break up tasks so you can stay busy while waiting.
I can’t speak for the Mozilla build, but in general I find compilation is almost always constrained only by CPU. Throwing extra memory or faster disk doesn’t help all that much…
If you build often, what helps most is RAM, because when you have enough of it, everything fits in RAM and is much faster to access: source tree, mercurial metadata, ccache, object files. With an i7-870 with 16GB RAM, I can build from scratch (rm -rf objdir) in a bit more than 1 minute (with almost no disk access). Add an SSD to the equation if you want cold cache builds (the first in the morning) to be fast too.
The Head of Engineering should be FIRED! A compile on a local machine requires 2 minutes? and a Link Library is made in X time? So how many Libraries are there Mr. Engineer?vToo much time is being consumed on source validation or revalidation… The machine times and people times are too LARGE to be excused. Hierarchy has been missed Mr. Engineering Dept and over paid when it consumes the community! KISS…
I pray you forwarded it.
@bigklipper – best comment ever.
post your mozconfig, and your actual machine specs. I know you bought it 2.5 years ago, but how much did it cost? 🙂
Sayre: interesting bits from /proc/cpuinfo:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz
stepping : 10
cpu MHz : 1998.000
cache size : 6144 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
Plus it has 4GB RAM, though only 3.6GB is usable due to some weirdness with the motherboard.
Sayre: oh, you asked for mozconfig as well. Something like this for a 64-bit debug build:
ac_add_options –enable-debug
ac_add_options –enable-optimize
ac_add_options –enable-tests
mk_add_options MOZ_MAKE_FLAGS=”–quiet -j2″
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/d64
Nothing too exciting. I wonder how much –enable-tests hurts. I’ve also tried ccache but found that it speeds up the rare case (remove build dir, rebuild) while hurting the normal cases quite noticeably, eg. a build from scratch takes 35 instead of 25 minutes. Maybe my lack of RAM is hurting there as well.
I’m still puzzled by the question of whether configure is supposed to run every build, and why sfink’s “do ‘make’ in the build dir” trick is so effective.
On Fedora, I can turn on gold with |sudo update-alternatives –config ld| (or manually change the /etc/alternatives/ld symlink to point to /usr/bin/ld.gold).
Get a spare desktop box and use it as a distcc server. I have one at home and at work, and run a cron job to automatically switch to using whichever one is accessible on the LAN I’m connected to.
I have a miniature makefile ~/mf that I use to encode the directories to rebuild, and make -f ~/mf to rebuild. It uses |ROOT := $(shell hg root)…| to come back into the tree containing my working directory. But that doesn’t help your problem of touching things outside of it. I have a similar problem — if I change an IDL file *within* that set of directories, it still requires a full rebuild, and it breaks in subtle ways until I do.
Maybe you should write a mini-Makefile like mine, but additionally have it run |hg diff | diffstat -l -p1| and automatically check that all of your changed files are within the expected directories.
Ooh. That seemed like a fun little challenge, so I wrote one. Save http://people.mozilla.org/~sfink/mozmk to ~/mozmk, |alias mk=”make -f ~/mozmk -j16 -s”| and just run |mk| when you want to rebuild.
The hg diff introduces a noticeable pause, and it doesn’t take applied patches in your queue into account. If the latter bothers you, install my mqext extension from ‘hg clone https://bitbucket.org/sfink/mqext‘ and use ‘hg touched -a | cut -f2’ in addition to hg diff to cut the time to a fraction of a second and have it look at applied patches as well. Something like ‘( hg touched -a | cut -f2; hg diff | diffstat -l -p1 ) | perl…’ ought to work.
Use –enable-shared-js for those happy times when you *can* get away with only modifying js/src.
If you leave your computer on all night, set up a cron job to make -k all of your object dirs during the night. Even if a build is useless, it’ll keep your ccache relatively up to date. For best results, build your “active” repo last. (That’s too much trouble for me.)
Consider setting CCACHE_BASEDIR. Then after you get pissed off that it messes up gdb source finding, implement the obvious hack to rewrite the files pulled from the cache. Instead of just turning it off, like I’m about to do.
You claimed that it reruns configure every time. Isn’t that only if you run make -f client.mk? If you make from the top level of the obj dir, it doesn’t do that unless it decides it needs to.
If you’re switching between a bunch of trees, and you may have to to hide the build latency, set up a cron job to run |hg relink| on all of them. If they aren’t all pointing to one “master” repo, set a ‘default-relink’ path in $REPO/.hg/hgrc to point to one of them. It might help a little in keeping the caches warm.
Use the try server to build for you, then download and run the builds locally. Just get your try syntax right. Only helps if you want builds with funky options (eg opt, or –enable-njn-hack17, or whatever) since they’ll take significantly longer than a single local build.
Use cgroups under Linux to prevent your builds from bogging down your machine, especially during linking. When I remember, I make one group for my browser, one for my builds, and let everything else fight it out.
I didn’t know you could do that! All the build instructions tell you to do “make -f client.mk” from the source dir. That only took 1 minute for a zero-change rebuild instead of the usual 3 minutes!
make -f client.mk doesn’t run configure if it doesn’t need to.
glandium: it does for me, it always has.
How do you get a single-change rebuild down to 3 or 4 minutes? For me it takes more than half an hour running make in the root of the object dir (Windows, 3.5 GB RAM). I know you can do it if you make in a subdirectory, but it is not always enough to make in a parent directory of the files you changed, and figuring out which directories to make in is quite impossible for me.
Jesper: apparently Windows is the slowest platform to compile on, and Linux is the fastest, due to toolchain differences. That would be part of the reason for your slowness.
Other than that, maybe it comes down to hardware differences. How old is your machine? Do you have 2 cores? What L2 cache do you have? I believe Win32 user space can only utilize 2GB of RAM, that the kernel uses the other 2GB, so that’ll hurt too.
If you use –enable-chrome-format=symlink then you don’t even need to rebuild after making a (non-preprocessed) JS UI change. (Non-preprocessed JS components are already symlinked.) You might still need to use -purgecaches (or I think there’s an equivalent environment variable).
Also, this can make some things faster:
http://mercurial.selenic.com/wiki/InotifyExtension
@nnethercode: you may want to take a look as to why it does.
python build/pymake/make.py -d -f client.mk MOZ_MAKE= 2>&1 | grep Remaking
4 cores and 8GB RAM speeds up my builds and keeps my machine responsive during linking (Windows 7).
I know that if I modify layout without changing public interfaces, I can just “pymake -sC obj-ff-debug/layout && pymake -sC obj-ff-debug/toolkit/library”. Ditto for gfx, content and other modules.
I think you have figured out one of the most important points: Code as much as you can in JS/XUL and as little as possible in C++ 😉
If you write your js code as an extension, use a file link from your profile to your source file, and set about:memory as your home page, then all you have to to is restart the browser for JS.
But you should do as much as possible in HTML not XUL. Then you can have a reload button on your browser and not even start it, just reload your js.
4GB of memory is really pushing it; 8GB helps a lot, and make sure
it’s DDR3, not DDR2. Even 8 isn’t enough; now I wish I’d gotten 12 or
16. Also 4 cores are better than 2, and real cores are a lot better
than hyperthreaded ones — I find a hyperthreaded core is worth about
60% to 65% of a real one in compute bound tasks. Real cores cost more
in energy though. Probably a big L2 is useful too; your 8400 is good
on that front.
Linking is slow but not catastrophic; it may be that your 4GB
exacerbates this. I hear Gold is way better but haven’t tried it. If
V doesn’t play nice with Gold, that’s definitely something we should
fix. I’m kinda surprised we haven’t had any bug reports of V+Gold
problems from the general V user population, though.
Core 2 Duo is slow. 8GB with lots of new cores and an SSD will get your builds really fast.
Some tips:
1. Build on Linux
2. Use ccache and -j N for large N, something like
export CCACHE_HARDLINK=1
export CC=”ccache gcc”
export CXX=”ccache g++”
mk_add_options MOZ_MAKE_FLAGS=-j4
Otherwise, I think the best thing is to not compile a lot. Debugging by constantly rebuilding with a small changes is not that effective, instead adding a lot of logging code once is better, etc. Obviously though this only works in some cases.
azakai: I’ve tried ccache on two separate occasions now, and it only slows things down for me. For me, the common case is rebuilding after making small changes. ccache doesn’t help at all with that — the files being built have by definition changed and so aren’t in the cache — but it does add a significant overhead to every compilation. Am I missing something? Everyone else seems to think ccache is hot stuff.
> ac_add_options –enable-optimize
Doesn’t that add “-O2” ? It’s a significant time-waster compared with “-O” and you don’t get much more performance for the added compilation time. I often use
ac_add_options –enable-optimize=”-g -O”
You could also try the fabled Clang, if you like to live dangerously.
Julian, you can use “ac_add_options –enable-debug-symbols” now to get debug symbols.
–enable-debug-symbols is turned on by default now, as far as I know. For my own purposes, I’ve recently written a python script into which I pass the directories containing files that I’ve modified, and it spits out a full pymake incremental command line that magically builds the correct directories to get a working build. Of course I keep running into edge cases, so it’ll probably just end up turning into a hardcoded duplication of the makefile dependencies.
My machine is a couple of years old laptop with 32 bit Windows, Intel P8600 (2 cores, 3M L2) and 4 GB RAM, of which I think about 3.5 GB can be used by applications.
Does –enable-chrome-format=symlink work on Windows? Does it work for tests also?
Be careful with ccache hardlink if you have multiple trees! If you do this, you may end up relinking more often than you have to.
See http://www.mail-archive.com/ccache@lists.samba.org/msg00587.html
> If V doesn’t play nice with Gold, that’s definitely
> something we should fix.
I believe the problem is libffi, and we looked into this before? I’ll dig up the mail.
> How do I enable gold?
On Ubuntu, it’s apt-get install binutils-gold
> I’ve also tried ccache but found that it speeds up
> the rare case (remove build dir, rebuild) while
> hurting the normal cases quite noticeably, eg. a
> build from scratch takes 35 instead of 25 minutes.
That’s pretty bad. I’ve hacked on ccache, but I didn’t put much work into the cache miss case. I’ll poke around and see if I can improve things, because there’s really no reason for it to add 10 minutes to a build.
Regarding SSDs, I re-profiled on my Linux box. It appears that moving from a magnetic drive to an SSD takes build times from 20m to 10m (with ccache enabled, cache directory on same disk as build, cache cleared). That’s surprising, since I recall builds taking ~12m before I got the SSD. But apparently ymmv.
Actually, nevermind about the SSD. The problem was that I didn’t move the ccache directory off the SSD initially.
On Linux, with both ~/.ccache, the source directory, and the objdir on the same disk, the build takes just as long with or without SSD. Then again, this machine has 12g of RAM.