Over the past week or so, there have been a few conversations on #developers about the ever-increasing time it takes to compile Firefox. Compiling Firefox used to be a relatively quick affair (well, on my machine, at least) and I’ve grown dissatisfied recently with builds taking ever-longer. All of the aforementioned discussions have boiled down to “well, we’re adding a lot of code and all that code takes longer to compile.”
Surely that can’t be the whole story. A mozilla-central build from a day or two took about 24 minutes on my machine (Linux x86-64 hyperthreaded 8-core @ 2.6GHz, srcdir on SSD, objdir on mechanical hard drive). We are not compiling 2.4x times more code than we were a year ago…or are we? Let’s look at some numbers.
I looked at my mozilla-central repository for some convenient tags to pick from and found the FIREFOX_AURORA_${REV}_BASE set of tags. I started compiling with mozilla-central tip, and then with FIREFOX_AURORA_24_BASE, and resolved to go as far back as I could without undue fiddling with things that wouldn’t compile. I made it to FIREFOX_AURORA_14_BASE before hitting compilation errors with moz_free. (19 and previous required small fiddling with js/src to set the right PYTHON, but that was easily scriptable.) That gives me 11 markers covering over a year of development.
I didn’t try compiling things multiple times to see how much the numbers fluctuated. I didn’t attempt to control for operating system disk caches or anything like that, just hg up -r $TAG, generated configure, ran configure --enable-optimize --disable-debug --disable-gstreamer (I haven’t bothered to get my gstreamer dependencies right on my development box) in a newly-created objdir, and recorded time for make -srj20. The compiler was GCC 4.7 from Debian stable.
m-c tag | real | user | sys |
15 | 13m2.025s | 83m36.229s | 5m51.238s |
16 | 13m40.359s | 87m56.302s | 6m15.435s |
17 | 12m58.448s | 94m21.514s | 6m35.121s |
18 | 14m21.257s | 112m42.839s | 7m36.213s |
19 | 15m8.415s | 120m48.789s | 8m8.151s |
20 | 16m26.337s | 137m34.680s | 9m11.878s |
21 | 18m55.455s | 165m29.433s | 10m45.208s |
22 | 19m27.887s | 185m3.242s | 11m53.853s |
23 | 21m9.002s | 203m46.708s | 12m51.740s |
24 | 21m51.242s | 206m49.012s | 12m55.356s |
tip | 24m37.489s | 219m27.903s | 13m47.080s |
Those 10-minute clobber build times were apparently the result of warm disk cache or my dreams. Or perhaps they’re due to an older compiler; I only started using GCC 4.7 recently and I’ve done some tests that show it’s a fair amount slower than GCC 4.4. Or maybe I was using a different build config then. Whatever the reason, we can see that compilation times have been increasing steadily, but can we correlate that to anything?
Let’s look at the number of C++, C, WebIDL, and IPDL source files. We include the last two because the conversion to WebIDL has been cited as a reason for increasing compilation times and IPDL because the IPDL compiler generates a number of source files as well (roughly three source files for every IPDL file). The C++ source files are split by extension; .cpp is the usual prefix to use inside Gecko, so separating .cpp and .cc gives some idea of how much outside C++ source code is being compiled. (Just counting source files is subject to issues, since we don’t compile everything in widget/ for every platform, but we’re counting all the source files therein as if they were. This is a rough estimate.)
m-c tag | cpp | cc | c | webidl | ipdl |
15 | 4060 | 577 | 2503 | 32 | 206 |
16 | 4117 | 1273 | 2893 | 34 | 213 |
17 | 4173 | 1278 | 2895 | 38 | 221 |
18 | 4468 | 1405 | 3123 | 61 | 227 |
19 | 4498 | 1408 | 3118 | 78 | 228 |
20 | 4586 | 1418 | 2779 | 169 | 228 |
21 | 4605 | 1392 | 2831 | 231 | 228 |
22 | 4979 | 1392 | 2980 | 305 | 228 |
23 | 5072 | 1393 | 2982 | 344 | 231 |
24 | 5101 | 1330 | 2983 | 413 | 234 |
tip | 5211 | 1427 | 3029 | 436 | 234 |
First things first: we have a lot of code in the tree.
Now, there’s a couple of interesting things to point out. While there’s a lot of IPDL source files, the increase in IPDL can hardly be held responsible for the increase in compile time. So that’s one suspect out of the way. From 16 to 17, we barely increased the amount of code, but build times went down. This change might point to the recorded times being subject to quite a bit of variability.
We added a lot of WebIDL files from 20 to 21 while holding everything else constant and build times went up quite a bit! But we added roughly the same number of WebIDL files from 21 to 22 along with a lot more code and while build times went up, they didn’t go up by the same amount: 28 minutes of user time going from 20 to 21 and ~20 minutes of user time going from 21 to 22. There are a number of reasons why this could be so: we might not be compiling all those 500 C/C++ source files we added in 22 (ICU landed off by default in 22, so that’s ~400 files right there), the WebIDL files added in 21 might have had more of an impact, compilation-wise, the timings might fluctuate quite a bit, and so on and so forth. And notice that we added the same number of WebIDL files again from 23 to 24 and build times hardly changed. Sure, WebIDL means more files to compile. But I don’t think they are adding a disproportionate amount of time.
And finally, source code file counts. We had about 7800 source code files to compile back at Aurora 15 (remember, IPDL counts as three). We have about 10600 source files to compile now on trunk. That’s about a 40% increase that corresponds to roughly a doubling in time to compile. And yes, there’s also a lot of JS and other things that have been added that increase our time to compile. But I don’t think the amount of source code is the whole story.
What about an increasing usage of templates? Those are particularly hard on the compiler.
Indeed. Templates have been cited as something that hits the linker particularly hard, since all the object files become much bigger and the linker winds up throwing a lot of code away. This effect is probably responsible for js/src/ becoming somewhat slower to build in the last several months. But I don’t know a good way to measure “increasing usage of templates”; perhaps by trying to correlate symbols in object files with template usage?
I haven’t heard of templates being a problem for the linker before. At any rate, I suspect one way to detect this is to build up a little database of compile time per compilation unit and then taking note of how that changes over time.
Oh, by the way: http://en.wikipedia.org/wiki/C%2B%2B0x#Extern_template
From a ‘getting by’ perspective, does ccache provide any sort of bandaid for the build? I’m assuming a reasonable amount of the source code doesn’t change from day to day.
I don’t know. I’ve never fiddled with getting ccache set up.
ccache helps, but very little in the general case. As soon as you update your tree you have to recompile most of it, because our dependency tree is very connected.
Is it possible to untangle the dep tree a bit? It surely a win if this reduces (cached) compile times.
ccache is trivial to set up in most distributions: just install it, and put /usr/lib/ccache in your $PATH. Done; you should find that rebuilds go far faster. Give it a try.
Also, while ccache doesn’t help greatly after updates, it does work quite well when doing incremental development. Header file changes that affect the preprocessed source will invalidate the cache, but that doesn’t affect the entire tree unless it’s one of the more common headers.
Well from 16 to 17, user time did go up, but real time went down. So ye, the build system somewhat improves parallelism.
Maybe mach or pymake can be used to measure which commands/files/directories are using that much time?
That’s a good point. I did land some patches in the 17 timeframe to build dom/ and netwerk/ in parallel; maybe those helped more than I realized.
Note that the JS engine used to be mostly C code, and is now mostly C++ code. The transition surely accounts for a lot.
Is it possible to precompile/pregenerate the *DL files (so if you don’t change any or only a few, you don’t have to process them all again)? If this is not a lot of effort to implement, it may be interesting to see some numbers if this can reduce compile time significantly for most compilation cases – every bit counts!
The generation scripts already create the code in memory, but only write the relevant file(s) if the contents of the file would change. That doesn’t change full build times, but it helps out quite a bit with incremental build times.
I ran some statistics on those numbers. The correlation between each of the 5 measurements and the user time are:
cpp 0.97
cc 0.24
c 0.14
webidl 0.98
ipdl 0.67
So increase in cpp and in webidl each independently are linearly correlated with almost all the increase in user time. Of course this doesn’t prove causation, just correlation. But if there is a way to not build webidl files for example, that could be a quick way to see if that has an effect here.
I also ran a least-squares minimization on them all (find coefficients so that a linear combination of the 5 measurements results in the user time), and it also gives large numbers for cpp and webidl (2.57 and 12.89, but those are not normalized so ignore the magnitudes). c and ipdl had negative coefficients, further showing they might not be related to the growth in user time.
Has anyone profiled the webidl and ipdl compilers to see if they have any obvious room for optimization?
Is it time for a “do we compile fast yet” site?
The webidl compiler has been profiled before and some nice speedups have been made. I mean, it’s important that it be fast, but we also have to compile 400+ generation C++ files, so…
I don’t know if anybody’s devoted time to the ipdl compiler. I ran it through Python’s profiler once and the work looked pretty scattered, no obvious hotspots.