{"id":499,"date":"2019-10-28T13:11:38","date_gmt":"2019-10-28T17:11:38","guid":{"rendered":"http:\/\/blog.mozilla.org\/nfroyd\/?p=499"},"modified":"2019-10-28T13:11:38","modified_gmt":"2019-10-28T17:11:38","slug":"evaluating-bazel-for-building-firefox-part-1","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/nfroyd\/2019\/10\/28\/evaluating-bazel-for-building-firefox-part-1\/","title":{"rendered":"evaluating bazel for building firefox, part 1"},"content":{"rendered":"

After the Whistler All-Hands<\/a> this past summer, I started seriously looking at whether Firefox should switch to using Bazel<\/a> for its build system.<\/p>\n

The motivation behind switching build systems was twofold.\u00a0 The first motivation was that build times are one of the most visible developer-facing aspects of the build system and everybody appreciates faster builds.\u00a0 What’s less obvious, but equally important, is that making builds faster improves automation: less time waiting for try builds, more flexibility to adjust infrastructure spending, and less turnaround time with automated reviews on patches submitted for review.\u00a0 The second motivation was that our build system is used by exactly one project (ok, two projects<\/a>), so there’s a lot of onboarding cost both in terms of developers who use the build system and in terms of developers who need to develop the build system.\u00a0 If we could switch to something more off-the-shelf, we could improve the onboarding experience and benefit from work that other parties do with our chosen build system.<\/p>\n

You may have several candidates that we should have evaluated instead.\u00a0 We did look at other candidates (although perhaps none so deeply as Bazel), and all of them have various issues that make them unsuitable for a switch.\u00a0 The reasons for rejecting other possibilities fall into two broad categories: not enough platform support (read: Windows support) and unlikely to deliver on making builds faster and\/or improving the onboarding\/development experience.\u00a0 I’ll cover the projects we looked at in a separate post.<\/p>\n

With that in mind, why Bazel?<\/p>\n

Bazel advertises itself with the tagline “{Fast, Correct} – Choose two”.\u00a0 What’s sitting behind that tagline is that when building software via, say, Make<\/a>, it’s very easy to write Makefile<\/code>s in such a way that builds are fast, but occasionally (or not-so-occasionally) fail because somebody forgot to specify “to build thing X, you need to have built thing Y”.\u00a0 The build doesn’t usually fail because thing Y is built before thing X: maybe the scheduling algorithm for parallel execution in make<\/code> chooses to build Y first 99.9% of the time, and 99% of those times, building Y finishes prior to even starting to build X.<\/p>\n

The typical solution is to become more conservative in how you build things such that you can be sure that Y is always built before X…but typically by making the dependency implicit by, say, ordering the build commands Just So, and not by actually making the dependency explicit to make<\/code> itself.\u00a0 Maybe specifying the explicit dependency is rather difficult, or maybe somebody just wants to make things work.\u00a0 After several rounds of these kind of fixes, you wind up with Makefile<\/code>s that are (probably) correct, but probably not as fast as it could be, because you’ve likely serialized build steps that could have been executed in parallel.\u00a0 And untangling such systems to the point that you can properly parallelize things and that you don’t regress correctness can be…challenging.<\/p>\n

(I’ve used make<\/code> in the above example because it’s a lowest-common denominator piece of software and because having a concrete example makes differentiating between “the software that runs the build” and “the specification of the build” easier.\u00a0 Saying “the build system” can refer to either one and sometimes it’s not clear from context which is in view.\u00a0 But you should not assume that the problems described above are necessarily specific to make<\/code>; the problems can happen no matter what software you rely on.)<\/p>\n

Bazel advertises a way out of the quagmire of probably correct specifications for building your software.\u00a0 It does this—at least so far as I understand things, and I’m sure the Internet will come to correct me<\/a> if I’m wrong—by asking you to explicitly specify dependencies up front.\u00a0 Build commands can then be checked for correctness by executing the commands in a “sandbox” containing only those files specified as dependencies: if you forgot to specify something that was actually needed, the build will fail because the file(s) in question aren’t present.<\/p>\n

Having a complete picture of the dependency graph enables faster builds in three different ways.\u00a0 The first is that you can maximally parallelize work across the build.\u00a0 The second is that Bazel comes with built-in facilities for farming out build tasks to remote machines<\/a>.\u00a0 Note that all build tasks can be distributed, not just C\/C++\/Rust compilation as via sccache<\/a>.\u00a0 So even if you don’t have a particularly powerful development machine, you can still pretend that you have a large multi-core system at your disposal.\u00a0 The third is that Bazel also comes with built-in facilities for aggressive caching of build artifacts<\/a>.\u00a0 Again, like remote execution, this caching applies across all build tasks, not just C\/C++\/Rust compilation.\u00a0 In Firefox development terms, this is Firefox artifact builds done “correctly”: given appropriate setup, your local build would simply download whatever was appropriate for the changes in your current local tree and rebuild the rest.<\/p>\n

Having a complete picture of the dependency graph enables a number of other nifty features.\u00a0 Bazel comes with a query language for the dependency graph<\/a>, enabling you to ask questions like “what jobs need to run given that these files changed?”\u00a0 This sort of query would be valuable for determining what jobs to run in automation; we have a half-hearted (and hand-updated) version of this in things like files-changed<\/code> in Taskcluster job specifications.\u00a0 But things like “run $OS<\/code> tests for $OS<\/code>-only changes” or “run just the mochitest chunk that contains the changed mochitest” become easy.<\/p>\n

It’s worth noting here that we could indeed work towards having the entire build graph available all at once in the current Firefox build system.\u00a0 And we have remote execution and caching abilities via sccache, even moreso now that sccache-dist is being deployed in Mozilla offices.\u00a0 We think we have a reasonable idea of what it would take to work towards Bazel-esque capabilities with our current system; the question at hand is how a switch to Bazel compares to that and whether a switch would be more worthwhile for the health of the Firefox build system over the long term.\u00a0 Future posts are going to explore that question in more detail.<\/p>\n","protected":false},"excerpt":{"rendered":"

After the Whistler All-Hands this past summer, I started seriously looking at whether Firefox should switch to using Bazel for its build system. The motivation behind switching build systems was twofold.\u00a0 The first motivation was that build times are one of the most visible developer-facing aspects of the build system and everybody appreciates faster builds.\u00a0 […]<\/p>\n","protected":false},"author":320,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[72457,153,5],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/posts\/499"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/users\/320"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/comments?post=499"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/posts\/499\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/media?parent=499"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/categories?post=499"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/tags?post=499"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}