when an implementation monoculture might be the right thing

It’s looking increasingly likely that Firefox will, in the not-too-distant future, build with a single C++ compiler across the four major platforms we support.  I’m uneasy with this, but I think I’ve made my peace with it, partly as a result of writing the piece below.

Firefox currently builds with three major C++ compilers across four platforms: Microsoft’s Visual C++ compiler (MSVC), GCC, and Clang.  A fair amount of work has been done to deal with peculiar bugs in all three compilers: you can go search the source code and/or Bugzilla to find hacks that were needed for one reason or another.  A fair amount of work has also been stalled or shelved because one or two compilers don’t quite measure up in some required area (e.g. standards support).  As you might imagine, many a Firefox engineer has bemoaned the need for cross-compiler compatibility.

Cross-implementation compatibility is something that Mozilla expends a lot of effort on in a different context.  We have a Tech Evangelism bugzilla component for outreach to sites who use techniques that don’t translate across browsers.  When new sites appear that deliberately block Firefox (whether because the launch team took the time to test with Firefox and determine the user experience wouldn’t be acceptable, or because cross-browser compatibility was an explicit non-goal), Firefox engineers go find the performance cliffs and fix them.  Mozilla has a long-history of promoting the benefits of multiple implementations of the web platform; some of the old guard might remember “Works best in all browsers” campaigns and the like.  If you squint properly, you can even see this promotion in the manifesto (principles 2, 5, 6, 7, and 9, by my reckoning).

So as nice as a single implementation might be, dealing with multiple implementations was a fact of life in building an high quality open-source browser.  We dealt with it, because it seemed like we would always need to support MSVC; who would invest the time to create an open source, MSVC-compatible compiler?

Well, Google, mostly, and a host of other people, because the past several releases of Clang have included an MSVC-compatible frontend, clang-cl.  (Indeed, Firefox has been using clang-cl for Windows static analysis builds for some time.)  And now that we have a usable non-MSVC compiler on Windows, we can contemplate using an open-source compiler to create our release Windows builds.  And once we have that, we can consider using (and potentially only supporting) a single compiler (Clang) for all of the major platforms we support; Linux would be the remaining holdout.  (Chrome already ships on Windows with clang and requires clang everywhere, FWIW.)

We might continue to require that things build with MSVC and GCC on relevant platforms, even if we’re not shipping these builds; even if this happened, such builds seem unlikely to last for very long, for all the reasons that we wanted them dropped in the first place.  I imagine we’d probably continue to accept patches to make things build with non-Clang compilers, as long as the patches were not intrusive, just like we accept patches for non-tier 1 platforms.

Supporting a single compiler has a number of advantages:

  • Cross-language LTO (i.e. inlining) between Rust and C++ (we could, of course, do this today, but we wouldn’t get the win on all platforms);
  • Mozilla engineers can fix bugs in Clang/LLVM if need be;
  • Fixes can be more easily backported from the Clang/LLVM development tree;
  • Contributors have fewer compiler quirks to hold up their patches;
  • Integrating and/or upgrading local copies of upstream projects becomes easier;
  • Performance tuning becomes somewhat more straightforward when you have a single compiler to worry about.

I am probably forgetting some along the way.  (I don’t think it’s true that we’ll be able to entirely eliminate hacks to pacify the compiler; you push on C++ hard enough and long enough, and you find yourself doing all manner of unusual things.  We might even find ourselves doing more hacks, since we can justify it via, “Since we can/can’t rely on the compiler to do X…”)

I can see all the advantages.  I can even admire the sheer coolness of some of them; cross-language inlining sounds fantastic!  But the analogy between the Web situation and the C++ compiler situation makes me uneasy: we ask web developers to write cross-browser compatible websites, with all the time and energy that requires.  We tout the goodness of supporting multiple implementations of the web platform.  However, in the implementation of that web platform, we are in the process of deciding that the benefits of supporting a single C++ implementation are greater than whatever benefits (engineering, philosophical, etc.) might accrue from supporting multiple implementations.

To be explicit: we are making the exact style of decision that we ask web development teams not to make.

After having proposed this and thought about it for a while, I think the analogy is a bit strained.  We make the argument that websites should be cross-browser compatible because we support the freedom of users to access those sites with whatever browser they like.  Whereas Firefox engineering is the only “consumer” of the compiler(s), and so we should optimize for that single consumer.  Indeed, we don’t really concern ourselves with cross-engine compatibility for the JavaScript that lies behind our UI.  Firefox users (generally) don’t care too much what compiler gets used to build Firefox, and they’d probably support a switch to a compiler monoculture if that meant the browser got faster!

(I’m not completely at ease with calling the two situations dissimilar; it’d be all too easy for a website to say they only care about a single “user”, viz. users of $BROWSER, and dispense with cross-browser support.  I want to have a stronger argument for this case, but I don’t at the moment…)

At the end of the day, I think I’m mostly in support (0.6 on the Apache voting scale?).  I think it will be cool when it’s done, and I will probably wind up doing some work in support of the project.  But I can’t completely shake my uneasiness.  What do you think?

Tags: , , ,

21 comments

  1. Robert O'Callahan

    Nice post.

    The problem with a browser monoculture is that it gives that browser vendor too much power over the Web. The problem with a C++ compiler monoculture is that it gives that compiler vendor too much power over C++. I think it’s pretty clear that the former is a much greater problem than the latter:
    * The latter really only affects software developers, the former affects everyone.
    * Control over the Web can be leveraged for many ends, e.g. make surveillance easier or harder, make the Web more or less friendly to different kinds of content, etc. It’s hard to see how control over C++ gives you much leverage over other things. Hardware architectures maybe?
    * The Web is the only widely deployed platform for distributing applications and content that isn’t controlled by a single vendor. C++ just isn’t that important.

    • Stephan Sokolow

      Agreed. I’d also argue that one vs. multiple C/C++ compilers on a single machine is much less bothersome than one vs. multiple browsers.

      I’m already used to running arbitrary build automation, which call arbitrary dependencies under the hood. It doesn’t really matter to me whether both make and setup.py and what have you will call the same compiler under the hood.

      Firefox and Chrome can’t combine into a single tabbed window with a single set of about:config tweaks and userChrome.css hacks, a single cookie store, the more featureful of the two extension APIs, Firefox’s ability to override site context menus with Shift+RightClick, etc.

      (Which, again, is about power balance. As an end-user, choosing between two open-source compilers doesn’t really matter to me… but a site forcing a specific browser on me cripples my ability to exercise choice in features.)

      Really, “requires browser X” is more like a less severe form of being forced to dual-boot between Windows and Linux.

    • Cameron Kaiser

      > C++ just isn’t that important.

      I think I’d agree with you more if it weren’t for the fact “the Web” (in the abstract sense) runs on browsers that are written in it.

  2. I’d also point out that there’s only one Rust toolchain too… if someone hypothetically created an independent implementation of the Rust language, would you put the effort into modifying the Firefox build scripts to work with the alternative? I suspect not… unless it offered some compelling advantages, whether performance or portability or other, it just wouldn’t be worth the effort of maintaining support for multiple implementations.

    So yeah, while monocultures aren’t ideal, they’re not without their advantages. In this case, I don’t think the disadvantages are big, so if it buys you a lot, go for it.

    • Johannes Löthberg

      Well, there’s mrustc already, and I vaguely recall reading about a third compiler a few months ago but can’t remember the name. On the other hand, it’s not claimed to be production-ready, but it is able to compile rustc.

      • Nathan Froyd

        mrustc is not suitable as a second Rust compiler currently, as it assumes that the Rust code is correct (i.e. has passed the borrow checker).

  3. I would also that that contributing to llvm/clang is trivial. The project very quickly gives w+ permissions (just write one or two patches, send an email and you get the rights). Contributions can be done on the ML or the Phabricator instance.

    gcc still requires a copyright assignments to the FSF ( https://gcc.gnu.org/contribute.html ), which can be done only by actual mail except in 3 countries ( https://www.gnu.org/prep/maintain/maintain.html#Copyright-Papers ). Contributions can only be done via the mailing list.

    Not mentioning how fast the clang testsuite is compared to the gcc.

    For MSVC, currently, it is not even possible…

    • > I would also that that contributing to llvm/clang is trivial. The project very quickly gives w+ permissions (just write one or two patches, send an email and you get the rights).

      Said like this, it sounds like anyone could “easily” pull a trusting trust attack…

  4. To riff on roc’s excellent points above, even if health of the C++ platform were as important as the health of the web platform the decision to move to a single compiler is not analogous to supporting a single browser in your web application in other ways. For one, end users don’t recompile Firefox every time they want to execute it, and the majority don’t run multiple browsers.

    When a website works only in browser X, users of that website are likely to use browser X full time. If an application only builds on compiler Z, then a developer probably installs a couple of compilers and lets the build systems sort it out.

    If browsers eventually innovate incompatibly, website developers will be forced to chose the most popular even if the less popular offers useful features. If compilers do that, developers can still chose the most useful one even if it’s less popular because they are the only direct consumers of the compiler.

    Even if somehow clang becomes the only C++ compiler in the world, developers can always choose different languages entirely. Innovation can still happen in the compiled app space. Replacing the web is an entirely different prospect.

  5. Rather Not Say Today

    This idea, along with the jump to embrace Slack, are bad ideas that could bite Mozilla.

    It’s true that embracing a compiler monoculture will have good short term payoffs for Mozilla. However, that’s true for everyone facing the browser support decision as well.

    Yes, there are bad effects for the web if everyone is Chrome-only. But consider it from their point of view: not only are there concrete benefits to this choice, but Mozilla itself supports the underlying logic.

    It’s also true that the web platform is more important than the C++ platform. However, this doesn’t matter to the folks making the actual decisions. In fact, maybe the opposite.

    The way I see it is, if you want to have high ideals — which you should — then you have to do the extra work to live through the consequences of them. Mozilla pays lip service to a lot of this.

  6. How many Makefile interpreters are out there that we use? How many Python interpreters? For that matter, how many Rust compilers are out there that we use? Rust was literally invented at Mozilla, and so we may suffer from a bit of “not-invented-here” syndrome with its young age. If I didn’t know better – and I don’t – I’d say parts of our ecosystem are already monocultures…

    This could be a very bad thing in the long run, but you also have to ask what competing tools for these languages exist.

    Thinking about monoculture and the risks it poses is very healthy. Letting it stop you from development isn’t necessarily.

    At best, it could be the start of pressure to the competitors to say, “Come on…” I think gcc and MSVC will be around for quite a while, because they have huge user bases beyond the browser space.

  7. Cameron Kaiser

    > they’d probably support a switch to a compiler monoculture if that meant the browser got faster!

    They’d probably support a switch to a browser monoculture if that meant websites got faster, too.

    This sounds more negative than I’d like it to sound, but this post sounds like someone’s made a decision already and people are gyrating wildly to not make it seem hypocritical. I doubt very much people are clamouring to make mozilla-central build with icc and tcc, but let’s not go backwards with a code base that already supports choices. This policy essentially implies we’re going to make gcc tier-3, despite the clear admission that clang is hardly perfect either.

  8. Could you support GCC-Clang duopoly as a compromise?

    I have no problem with ditching MSVC, which is proprietary, has awful C support, and is almost always the odd one out.

  9. To me it’s ridiculous to even compare the two situations as though they’re on even ground. If a web engine monoculture develops, nobody is going to step up and create a new engine anytime soon (even collectively). Heck, just maintaining WebKit or Gecko would be a challenge, let alone keeping them up to date with the web’s progress. We already lost Presto because of this.

    It’s comparatively trivial to write patches to keep a browser engine working with another compiler, compared to writing an entire browser engine. If people could make Firefox work with ICC, they can make it work with GCC or MSVC. I honestly can’t see that happening for an entire browser engine unless the monoculture was so fractured and unstable that it was unlikely to form in the first place.

  10. > Whereas Firefox engineering is the only “consumer” of the compiler(s), and so we should optimize for that single consumer.

    I don’t think this is true. Firefox code is built by many more people than just Mozilla. Linux distributions are probably the most prominent (but not sole) example. Some of them are invested in GCC and have GCC developers involved to quickly deal with possible issues when they arise.

    Firefox is an important (release-blocking) part of most Linux distributions. Not everyone is comfortable with expanding the set of super-critical components by another C++ compiler. For some, the rustc (and thus LLVM) requirement (while ultimately necessary) was hard enough to swallow.

    > I’m not completely at ease with calling the two situations dissimilar; it’d be all too easy for a website to say they only care about a single “user”

    Yeah, just like that.

  11. Marcel Kincaid

    “the analogy is a bit strained.”

    Quite an understatement. The analogy is completely bogus. clang provides a single language frontend that maps to backends for all target architectures. There’s no reason to support more than one such frontend. Supporting multiple browsers is nothing at all like that.

  12. Using a non standard compiler on any particular platform means that you lose a lot of support and tooling from your OS vendor. It also means that you will be walking down a lonely, lightly trodden path. Which is not what you want when shipping an important, popular binary.

  13. Jens Mander

    The company I work for considered a similar move. Funnily, it was mostly driven by the non-Windows developers. In the end, it didn’t happen because the toolchain is not ready and we can’t sell our users a 10% drop in performance and our developers a 15% increase in compile time: http://blog.llvm.org/2018/03/clang-is-now-used-to-build-chrome-for.html
    These days, MSVC is in good shape, I must admit.

    Anyway, I find it interesting that Mozilla, where diversity is promoted a lot, recognizes the merits of monocultures…

  14. Rodrigo Kochenburger

    In my view, the comparison is not entirely fair.

    The web standardization is an effort to have different browsers that adhere to the same specification and behavior. The analogous to C compilers would be all the versions of ANSI C, and doesn’t directly apply to the user of the compiler, the same way standardization of web does not require the user to worry about the compatibility. Instead the user, ideally, chooses one browser and that single browser will work for all the cases.

    I think the problem is that you’re looking from two different perspectives. Mozilla is not in the business of building compilers, and as a user shouldn’t be the one enforcing compatibility with the compilers. The compilers should be more compatible between them but that’s a different subject outside of the context of Mozilla (I think).

    Pick the best language and compiler that allows you to provide higher quality software with less bugs, and therefore better conformance to the web standards, which is really what matters to the product.

  15. So I actually think the Situation of Mozilla here with C++ developement environments is fairly similar to web developers and browsers, at least as far as technical factors go. In both cases there seems to be some good arguments that life would be easier if there was one implementation that has to be delt with instead of several. However I think competition, and diversity in design choices may well be good reasons to have multiple implementations. So as a technical matter I’m far from comfortable with it, but monocultures may be good. In any case it certainly seems like we’re headed that way for C++ whether or not you or I like it, but who knows.
    If I’m reading Roc’s comment correctly I think he’s arguing that a large part of the problem with a monoculture for the web would be a problem with giving control to one company, not necessarily with one implementation. Which I agree with, and is probably an important point here. However it does prompt the question how terrible would it be if we had one web browser that was controled by multiple companies. I suppose the answer there is the one time it was tried the two companies involved decided they’d rather have separate implementations.
    It seems to me its a little unfair to characterize people building Firefox as one user even ignoring Linux distros etc, I can’t think of as many examples off the top of my head, but compilers do have features and it seems as reasonable to for an individual engineer to prefer one compiler to another as for a user to prefer one browser to another. It would be closer to the web case to ask if Mozilla would be ok making internal sites only target Firefox since there only user is also Firefox engineering. To be clear I don’t know what I think of that, but it seems to me if one has technical merit then the other may well too. If the answer here is to add features to one compiler til it meets your needs as a technical matter that applies to the web too we could certainly add all the features of browsers A abd B to C and then say we care about browser c and everyone should switch to it because its better in every way.
    That said I’m not sure its great, but I’m certainly on board with getting rid of MSVC, and other proprietary compilers. As for going with the OS provided compiler there’s something to be said for that, but there’s also a lot to be said for using better tools and having control of those tools. A couple months ago I ran into a bug where fairly recent versions of Solaris studio’s linker couldn’t generate correct relocations into string literals, and I’ve run into a couple of silly preprocessor bugs in xlc. Then there’s the issue that those compilers will only ever support C++ 03. I can’t say there’d be no issues with using gcc to build for AIX and Solaris at $job, there probably would be some, but I think it would absolutely be worth moving toward compiling with gcc on those platforms, and that’s at least moving towards a monoculture.

  16. As GCC developer I naturally think limiting your set of supported compilers to clang is a bad idea. I am actively working on making GCC better for compilation of larger programs like Firefox and it is not hard to measure that GCC build binaries are smaller and faster in general. Text segment of GCC 8 built binary is 67MB and clang 6 build is 81MB and this gets more visible with LTO where GCC does 58MB and Clang 6 is 88MB-90MB (thin-lto wrt LTO). GCC also builds faster for me.

    There are number of bugs fixed due to GCC ODR warnings, sanitizers and LTO effort which would not be fixed otherwise.

    So I would rather prefer to see LTO enabled builds with GCC on Linux targets and hopefully also eventually Rust having GCC as backend option. This will keep the competition going.

    LTO was tested on talos and seems to generally improve performance https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&newProject=try&newRevision=7e5bd52e36fcc1703ced01fe87e831a716677295&framework=1&selectedTimeRange=172800