I got 99 problems…and compilation time is one of them

Over the past week or so, there have been a few conversations on #developers about the ever-increasing time it takes to compile Firefox.  Compiling Firefox used to be a relatively quick affair (well, on my machine, at least) and I’ve grown dissatisfied recently with builds taking ever-longer.  All of the aforementioned discussions have boiled down to “well, we’re adding a lot of code and all that code takes longer to compile.”

Surely that can’t be the whole story.  A mozilla-central build from a day or two took about 24 minutes on my machine (Linux x86-64 hyperthreaded 8-core @ 2.6GHz, srcdir on SSD, objdir on mechanical hard drive).  We are not compiling 2.4x times more code than we were a year ago…or are we?  Let’s look at some numbers.

I looked at my mozilla-central repository for some convenient tags to pick from and found the FIREFOX_AURORA_${REV}_BASE set of tags.  I started compiling with mozilla-central tip, and then with FIREFOX_AURORA_24_BASE, and resolved to go as far back as I could without undue fiddling with things that wouldn’t compile.  I made it to FIREFOX_AURORA_14_BASE before hitting compilation errors with moz_free.  (19 and previous required small fiddling with js/src to set the right PYTHON, but that was easily scriptable.)  That gives me 11 markers covering over a year of development.

I didn’t try compiling things multiple times to see how much the numbers fluctuated.  I didn’t attempt to control for operating system disk caches or anything like that, just hg up -r $TAG, generated configure, ran configure --enable-optimize --disable-debug --disable-gstreamer (I haven’t bothered to get my gstreamer dependencies right on my development box) in a newly-created objdir, and recorded time for make -srj20.  The compiler was GCC 4.7 from Debian stable.

m-c tag real user sys
15 13m2.025s 83m36.229s 5m51.238s
16 13m40.359s 87m56.302s 6m15.435s
17 12m58.448s 94m21.514s 6m35.121s
18 14m21.257s 112m42.839s 7m36.213s
19 15m8.415s 120m48.789s 8m8.151s
20 16m26.337s 137m34.680s 9m11.878s
21 18m55.455s 165m29.433s 10m45.208s
22 19m27.887s 185m3.242s 11m53.853s
23 21m9.002s 203m46.708s 12m51.740s
24 21m51.242s 206m49.012s 12m55.356s
tip 24m37.489s 219m27.903s 13m47.080s

Those 10-minute clobber build times were apparently the result of warm disk cache or my dreams. Or perhaps they’re due to an older compiler; I only started using GCC 4.7 recently and I’ve done some tests that show it’s a fair amount slower than GCC 4.4. Or maybe I was using a different build config then. Whatever the reason, we can see that compilation times have been increasing steadily, but can we correlate that to anything?

Let’s look at the number of C++, C, WebIDL, and IPDL source files. We include the last two because the conversion to WebIDL has been cited as a reason for increasing compilation times and IPDL because the IPDL compiler generates a number of source files as well (roughly three source files for every IPDL file). The C++ source files are split by extension; .cpp is the usual prefix to use inside Gecko, so separating .cpp and .cc gives some idea of how much outside C++ source code is being compiled. (Just counting source files is subject to issues, since we don’t compile everything in widget/ for every platform, but we’re counting all the source files therein as if they were. This is a rough estimate.)

m-c tag cpp cc c webidl ipdl
15 4060 577 2503 32 206
16 4117 1273 2893 34 213
17 4173 1278 2895 38 221
18 4468 1405 3123 61 227
19 4498 1408 3118 78 228
20 4586 1418 2779 169 228
21 4605 1392 2831 231 228
22 4979 1392 2980 305 228
23 5072 1393 2982 344 231
24 5101 1330 2983 413 234
tip 5211 1427 3029 436 234

First things first: we have a lot of code in the tree.

Now, there’s a couple of interesting things to point out. While there’s a lot of IPDL source files, the increase in IPDL can hardly be held responsible for the increase in compile time. So that’s one suspect out of the way. From 16 to 17, we barely increased the amount of code, but build times went down. This change might point to the recorded times being subject to quite a bit of variability.

We added a lot of WebIDL files from 20 to 21 while holding everything else constant and build times went up quite a bit! But we added roughly the same number of WebIDL files from 21 to 22 along with a lot more code and while build times went up, they didn’t go up by the same amount: 28 minutes of user time going from 20 to 21 and ~20 minutes of user time going from 21 to 22. There are a number of reasons why this could be so: we might not be compiling all those 500 C/C++ source files we added in 22 (ICU landed off by default in 22, so that’s ~400 files right there), the WebIDL files added in 21 might have had more of an impact, compilation-wise, the timings might fluctuate quite a bit, and so on and so forth. And notice that we added the same number of WebIDL files again from 23 to 24 and build times hardly changed. Sure, WebIDL means more files to compile. But I don’t think they are adding a disproportionate amount of time.

And finally, source code file counts. We had about 7800 source code files to compile back at Aurora 15 (remember, IPDL counts as three). We have about 10600 source files to compile now on trunk. That’s about a 40% increase that corresponds to roughly a doubling in time to compile. And yes, there’s also a lot of JS and other things that have been added that increase our time to compile. But I don’t think the amount of source code is the whole story.

17 comments

  1. What about an increasing usage of templates? Those are particularly hard on the compiler.

    • Nathan Froyd

      Indeed. Templates have been cited as something that hits the linker particularly hard, since all the object files become much bigger and the linker winds up throwing a lot of code away. This effect is probably responsible for js/src/ becoming somewhat slower to build in the last several months. But I don’t know a good way to measure “increasing usage of templates”; perhaps by trying to correlate symbols in object files with template usage?

  2. From a ‘getting by’ perspective, does ccache provide any sort of bandaid for the build? I’m assuming a reasonable amount of the source code doesn’t change from day to day.

    • Nathan Froyd

      I don’t know. I’ve never fiddled with getting ccache set up.

      • ccache helps, but very little in the general case. As soon as you update your tree you have to recompile most of it, because our dependency tree is very connected.

        • Is it possible to untangle the dep tree a bit? It surely a win if this reduces (cached) compile times.

      • ccache is trivial to set up in most distributions: just install it, and put /usr/lib/ccache in your $PATH. Done; you should find that rebuilds go far faster. Give it a try.

        Also, while ccache doesn’t help greatly after updates, it does work quite well when doing incremental development. Header file changes that affect the preprocessed source will invalidate the cache, but that doesn’t affect the entire tree unless it’s one of the more common headers.

  3. Arpad Borsos

    Well from 16 to 17, user time did go up, but real time went down. So ye, the build system somewhat improves parallelism.
    Maybe mach or pymake can be used to measure which commands/files/directories are using that much time?

    • Nathan Froyd

      That’s a good point. I did land some patches in the 17 timeframe to build dom/ and netwerk/ in parallel; maybe those helped more than I realized.

  4. Note that the JS engine used to be mostly C code, and is now mostly C++ code. The transition surely accounts for a lot.

  5. Is it possible to precompile/pregenerate the *DL files (so if you don’t change any or only a few, you don’t have to process them all again)? If this is not a lot of effort to implement, it may be interesting to see some numbers if this can reduce compile time significantly for most compilation cases – every bit counts!

    • Nathan Froyd

      The generation scripts already create the code in memory, but only write the relevant file(s) if the contents of the file would change. That doesn’t change full build times, but it helps out quite a bit with incremental build times.

  6. I ran some statistics on those numbers. The correlation between each of the 5 measurements and the user time are:

    cpp 0.97
    cc 0.24
    c 0.14
    webidl 0.98
    ipdl 0.67

    So increase in cpp and in webidl each independently are linearly correlated with almost all the increase in user time. Of course this doesn’t prove causation, just correlation. But if there is a way to not build webidl files for example, that could be a quick way to see if that has an effect here.

    I also ran a least-squares minimization on them all (find coefficients so that a linear combination of the 5 measurements results in the user time), and it also gives large numbers for cpp and webidl (2.57 and 12.89, but those are not normalized so ignore the magnitudes). c and ipdl had negative coefficients, further showing they might not be related to the growth in user time.

  7. Has anyone profiled the webidl and ipdl compilers to see if they have any obvious room for optimization?

    Is it time for a “do we compile fast yet” site?

    • Nathan Froyd

      The webidl compiler has been profiled before and some nice speedups have been made. I mean, it’s important that it be fast, but we also have to compile 400+ generation C++ files, so…

      I don’t know if anybody’s devoted time to the ipdl compiler. I ran it through Python’s profiler once and the work looked pretty scattered, no obvious hotspots.