Introducing the Azure project

Motivation

Firefox 4′s graphics performance is great, and some last-minute “low-hanging fruit” inefficiencies Robert O’Callahan found made us even faster on some demos in Firefox 5. We’re not content with “great”, though, and our investigations into how to make drawing even faster have revealed that some of our choices in Gecko’s graphics engine aren’t optimal for performance.

How Mozilla renders

To display a web page, Gecko’s layout engine looks at how that page is constructed, and divides it into independent layers. Each of those layers is drawn using Cairo, a 2D graphics API built on top of the operating system’s graphics APIs. Cairo’s drawing into layers can be hardware accelerated — this is known as “content acceleration” — or done in software. Right now, content acceleration is fastest on Windows Vista and 7, where we use Direct2D, but we want to be able to extend good acceleration to all operating systems, including mobile.

After drawing, layers are composited together using either the platform’s 3D APIs (OpenGL or Direct3D) or Cairo, and the result is displayed on the screen.

Format conversions

Cairo uses a fixed point representation of numbers. Unfortunately, this means that when Cairo calls get mapped to an API like Direct2D, which uses floating point numbers, two conversions are required: one from from your internal floating point representation to Cairo’s fixed point representation, in which it is initially stored, and then back to floating point again when Cairo makes Direct2D calls.

Statefulness

Cairo is a stateful API. This means that any given function call depends on the results of the previous function calls to determine its result. Unfortunately, statefulness strictly adds overhead to an intermediate graphics API: if it maps to another stateful API, Cairo has to track its own state separately from the underlying API (usually due to subtle differences between the two). Also, Cairo uses a stateless surface API internally; whether the underlying API is stateful or stateless, there’s always a conversion from one to the other. Further, the HTML5 Canvas element is a stateful API that’s built on top of Cairo, and it has to have its own state tracking too.

What makes this worse is that Gecko is better-suited by a stateless API; when we begin rendering any part of a web page, we need to reset most of Cairo’s stored attributes, since they’re determined by the element’s CSS.

What can we do about it?

We would very much like to remove these unnecessary conversions and state tracking. We’d also like to be able to hardware accelerate content drawing in an easy and cross-platform way.

Mozilla needs a stateless graphics API that’s closer both to platform APIs and hardware.

This API needs to be close enough to Direct2D to be little more than a thin wrapper, but general enough to be implemented on OpenGL and Direct3D 9.

The Azure project

To accomplish this, the Mozilla graphics team is creating a new 2D graphics API. It’s going to be stateless, and significantly closer to Direct2D. We’re also going to develop new “3D” backends for drawing content in an accelerated way to OpenGL, Direct3D 9, and Direct3D 10.

The backends

Once complete, the new 2D API will be implemented by several different backends, depending on OS, hardware, and driver. We will also have a fallback implementation, using Cairo, that we’ll use when we don’t have a more specialized backend available.

Direct2D

We don’t make the best use of Direct2D right now. Part of it is the floating/fixed point impedance mismatch I described earlier, but part of it is capabilities Cairo exposes that Direct2D doesn’t implement. These need to be worked around or emulated using software. The API we create as part of the Azure project will be close enough to Direct2D to not require fallbacks.

Quartz / CoreGraphics

On OS X, Cairo uses its Quartz backend for all drawing. Quartz can be fast: Safari is faster than Firefox on some demos, even though we both use the same backend for drawing. We believe one reason this is the case is because Quartz is stateful: Cairo needs to convert from its stateful API to its internal stateless surface API, then back to Quartz’s stateful API. We’ve designed Azure’s stateless 2D API so that it can map from stateless to stateful in a more optimal, and optimizable, way.

Cairo

Cairo is our failsafe implementation. It’s the backend we’ll use before others are ready or when we explicitly choose not to use hardware acceleration, for example because of driver issues. We also need Cairo for printing support.

It’s possible that we will decide to modify our internal copy of Cairo to allow us to use the internal stateless surface API directly, since we think Cairo’s surface API is a pretty good match to Azure’s 2D API.

3D Backends

The goal of the “3D” backends is to implement Azure’s 2D API on top of the common 3D APIs (OpenGL and Direct3D) so we can have content acceleration on Windows XP, OS X, Linux, and mobile. Our intent with these 3D backends is to be fully cross-platform, including Windows 7. This lets us better control our performance characteristics and bugs, rather than spending a lot of time and energy working around Direct2D bugs.

Electrolysis and Azure

The Azure 2D API will make it possible for us to hardware accelerate content drawing, even in the distant future when we “sandbox” our content processes. This new 2D API is required for hardware accelerated sandboxed content processes, but isn’t required for the initial Electrolysis roll-out.

Timeline and checkpoints

Over the past few weeks, we created a proof-of-concept Direct2D backend to test our hypothesis that we could create a faster non-Cairo API. Our hypothesis was correct — our backend is significantly faster than the Direct2D Cairo backend on some commonly-used Canvas demos and benchmarks. This work is all available in the graphics project branch.

Our initial success led to our current plan: to design and test this new 2D API as a backend for the HTML5 canvas element. The graphics team has included getting the Azure Direct2D canvas backend enabled in mozilla-central as a Q2 goal.

We are also in the middle of writing similar canvas backends for Cairo and Quartz, both of which are already in the graphics project branch, but those are not yet operational. We haven’t defined specific time goals for those backends.

Postscript: Why “Azure”?

Jim Clark invented the first GPU at Stanford in 1979, called the “Geometry Engine”. He founded SGI based on this work.

SGI made many workstations and servers with “colour” names like Onyx, Indigo, and Crimson. They are also the original inventors of OpenGL.

After SGI, Clark founded a little-known software company called Netscape.

It seemed only fitting to pay homage to him in this small way!

81 comments

  1. “This lets us better control our performance characteristics and bugs, rather than spending a lot of time and energy working around Direct2D bugs.”

    So are you looking to move away from Direct2D once the 3D backends are ready?

    • @Sid: We definitely want the option of moving away from Direct2D. Whether we actually do depends on performance, completeness, etc.

    • AFirefoxFan4Ever

      I hope that this API will help Firefox render content in an independent way from the CPU because even the HA inside firefox still cpu dependant

  2. This sounds really promising. I am a happy all time beta user for Firefox and hope to see my SVG demos get as fast as in Chrome, soon.

    Ciao, Juve

  3. Sound great. A few questions:

    “This API needs to be close enough to Direct2D to be little more than a thin wrapper” – does that mean Windows will always have an advantage over OS X, Linux and Android as Azure on Direct2D will have the least amount of overhead?

    How does this compare to Skia?

    Will this enable Native HTML5? :-)

    • Windows won’t have an advantage, except in that Direct2D already exists and we still have to write the 3D backends. We think the Direct2D API is a good one to emulate when we write our own 2D-on-3D rasterizer.

      We haven’t done any measurement of Skia yet.

  4. Please give us Windows XP SP3 users hardware accelerated OpenGL rendering for content rendering acceleration too, not just layers acceleration.

    • You’ll be happy to know that that is indeed the plan! :D (But we’re going to use Direct3D on Windows, most likely.)

      • Why not both? OpenGL is still best supported on Windows.

        • You’ll probably be able to use OpenGL on Windows, but we’ll use Direct3D by default. There’s no reason to use OpenGL when Direct3D drivers are of equal or better quality.

          • Reason is simple – supporting an open standard (OpenGL) instead of Direct3D. Mozilla usually has a strong stance on open standards. Understandably, there might be concerns of lacking or poor OpenGL drivers, vs. better supported Direct3D, but at least OpenGL should be available as an option, for those who prefer open standards.

          • We’ll certainly have some runtime controls for what 3D API is being used, but they’ll be hidden preferences (i.e., about:config), not surfaced in the UI.

          • Sounds good, thanks.

    • Wow. XP’s 10th birthday is coming up soon. Are you guys going to have a party to celebrate?

  5. Hi,
    I read this post quickly, as I did for the comments.
    AFAIK, your project shouldn’t be called “Azure” as this is a trademark from Microsoft for its cloud stuff thing.

    Regards

    • They trademarked the term “Windows Azure” and “SQL Azure” so just the name of the color “Azure” does not violate their trademark.

      • It is confusing to call it Azure because Microsoft is already publicizing that name for its own product that has nothing to do with this, and they have $millions advertizing budget. Counterproductive.

        • Maybe it should be written [azyr] so that these english speaking folks manage to pronounce it (somewhat) right. This would distinguish it from the Microsoft thing. When I saw a commercial for Windows Azure I thought “What the h*** is this Windows Escher? Why does Microsoft name a product after a famous artist?”, until I realized that english people cannot pronounce french words at all. (Also I heard one pronounce d’Artagnan something like “Dartainion”. lol)

        • I agree that Azure is confusing. For the longest time I kept thinking Microsoft, Microsoft… until I saw this article. It may not violate any trademarks but its sure confusing!

          Also I’m confused about Azure and Direct2D vs Azure and Direct3D. Wikipedia has only confused me. Why are your using Direct2D and not Direct3D? What are advantages / disadvantages of both approaches? What does IE9 use?

          • Sidharth

            “Wikipedia has only confused me. Why are your using Direct2D and not Direct3D? ”
            D2D is the newer product.

            “What does IE9 use?”
            D2D

            “why?”
            So they can sell windows7; xp doesn’t support D2D.

      • I agree with the uncertainty of naming this project Azure. I believe Microsoft is choosing that as their overall name for their “cloud initative” so that they will be naming some ten things “Azure (version of ____)”.

        Checking above FF appears just to be doing a fun little “shout out to Jim Clark’s penchant for colors”, so any color name will do.

  6. [...] about to dump Cairo They’re building their own 2d rendering api. It’s called… the Azure Project. Hopefully it’ll speed up Firefox even more! browsers firefox var [...]

  7. > distant future when we “sandbox” our content processes
    How “distant” are we talking about here?

    In any case, good stuff, though this is a massive undertaking. I don’t know how many rendering primitives you need, but subpixel-AA’d font rendering sounds like it’ll be quite a bit of work (moreso if you aim to match each platform’s rendering).

    • I don’t know if we even know how long it’ll be before we get sandboxed content processes. 1 year+, maybe.

      And yeah, this is a massive undertaking. We do hope/plan to use the platform renderer for font masks; how well that ends up working is another thing entirely. :)

  8. I agree with the suggestion of a name change, I realize the color is not trademarked, but it may be confusing due to Microsoft use of Windows Azure for cloud computing.

    • Don’t really think this will be an issue as this is not a consumer-facing project, but rather an internal, development project.

      • Hit #1 on Google is ‘Windows Azure’. God help anyone searching for Mozilla Azure related content, especially if it’s Windows specific.

  9. Great stuff! A very bold endeavour I would say. Good luck!

  10. Will Azure and its backends onlybe used for the canvas HTML5 canvas element or will they eventually replace cairo for all drawing tasks (in all layers)?

    • Initially Azure will only implement canvas backends, but in the long term our plan is to replace cairo for all drawing tasks.

  11. “This new 2D API *is* required for hardware accelerated sandboxed content processes, but isn’t required for the initial Electrolysis roll-out.”

    Does this mean that the first Electrolysis-based Firefox will not have HW acceleration?

    • The first Electrolysis-based Firefox will have the same type of hardware acceleration the previous version has, which is likely to be very similar to the acceleration in Firefox 4.

      (The reason I included this section is because there was some worry that we’d need the Azure project to be complete before we could enable Electrolysis.)

  12. I have been wondering about this in a big way since my biz is tied to browser differences. How is this going to change cross browser testing? Meaning are the different backends going to render differently in terms of layout… I would almost think they would… depends on where the render is at exactly. Is the layout engine separate? What I am hoping will not happen is that ff5 in direct2d will looking different than ff5 in opengl….
    Any thoughts on that?

    • Layout is, indeed, a separate module, and we have very extensive tests that make sure it stays the same, so except for some minor font rasterization differences, rendering should be exactly the same from one OS to the other. No need to worry!

  13. [...] This problem statement came from Mozilla as Joe Drew introduces the Azure project. [...]

  14. [...] the pesky memory-scrubbing process called garbage collection. And the graphics engine, called Azure, is intended to get along better with Windows’ graphics interfaces while still working with [...]

  15. [...] news, Mozilla also announced enhancements for the Gecko layout engine. A project called “Azure” focuses on the creation of a new 2D graphics API for Firefox as well as new 3D backends to [...]

  16. This means that Azure without a good GPU use Cairo backend? For example a Windows or Linux machine without good hardware/drivers have an advantage using Azure or not?

  17. [...] ready to move beyond the planning phase and begin coding. The new graphics core has been dubbed Azure, an homage to SGI innovator Jim Clark — who invented the first GPU in 1979. Azure’s aim is to [...]

  18. [...] new graphics core has been dubbed Azure, an homage to SGI innovator Jim Clark — who invented the first GPU in 1979. Azure’s aim is to [...]

  19. [...] entwickelt das Unternehmen unter der Bezeichnung Azure eine Grafik-Engine, die besser mit den Windows-Grafikschnittstellen zurechtkommen sollen, ohne dass [...]

  20. [...] und den Speicherverbrauch reduzieren. Zudem entwickelt das Unternehmen unter der Bezeichnung Azure eine Grafik-Engine, die besser mit den Windows-Grafikschnittstellen zurechtkommen sollen, ohne dass [...]

  21. [...] it all at once. Mozilla also announced enhancements for the Gecko layout engine in a separate post by developer Joe Drew. Project Azure, as it is known, focuses on the creation of a new 2D graphics API for Firefox [...]

  22. [...] also announced enhancements for the Gecko layout engine in a separate post by developer Joe Drew. Project Azure, as it is known, focuses on the creation of a new 2D graphics API for Firefox [...]

  23. Will it be an “independent” project (like cairo/skia)? Or will it be projected to be gecko only?

  24. Surely it would be better to add the features you need (stateless and floating point) to cairo?
    Then it would not just be Mozilla benefiting, but all the users of cairo.
    It would still be a large undertaking, but if cairo really wants to treat Windows as a first-class platform, they should be willing to accept such a change.

    • Robert O'Callahan

      It’s hard to make large changes to cairo because cairo makes stable API guarantees. We don’t need cairo API stability in most cases, but other cairo users do.

  25. [...] the pesky memory-scrubbing process called garbage collection. And the graphics engine, called Azure, is intended to get along better with Windows’ graphics interfaces while still working with [...]

  26. Isn’t it a bit short-sighted to *now* consider a cross-platform graphics API built around Direct*D when Windows installs are waning and the rest of the planet use OpenGL?
    I think my issue is azure being close enough to direct2d to cause (even minor) performance hits on opengl.

    • We think that Direct2D is a good API for general hardware acceleration of 2D drawing, which is why we want the Azure 2D API to be reasonably close to it. Direct3D and OpenGL are close enough to not make a difference w.r.t. implementation of a 2D API on top of it.

      Keep in mind that we eventually want to be able to choose between Direct2D and our Azure “3D” backends.

      • Will Azure have any relationship with OpenVG? From what I see it does much of the same things. Also OpenVG implementations are also built atop OpenGL.

  27. Oh please, pleae rename this project. Not only Microsoft has a project running with this name, but so do I.
    I’m about to develop a game called “Azure: Infinite Skies” (http://azure.tuxfamily.org) which name gets more and more confused :-S

  28. [...] 651858 – [Windows] Implement Direct2D Azure Backend and use it for <canvas>. (joe’s post, bas’s [...]

  29. [...] the content of a <canvas> element (in a 2D context). You can read a detailed explanation of the Azure project and its next steps on Joe Drew’s [...]

  30. Do you plan to make the 3D backend configurable for Azure? I.e. let’s say on Windows, where Direct3D and OpenGL are available both, would user be able to use OpenGL as Azure’s backend as an option? (This at least would provide an option of not depending on proprietary APIs).

  31. Hi!

    I quickly read your post and the comments so sorry if I missed a point but it seems to me you took the slippery road of rewriting yet another 2D API. Since you already relied on Cairo, wouldn’t it have been a huge + for Mozilla in terms of coding effort and a huge + for OSS in terms of code sharing and reuse, to help improve Cairo like preparing a future stateless version of Cairo’s API and contributing efficient backends… dunno, just wondering! Well, the Cairo folks can always look into Azure’s code later on anyway, can’t they?

    Cheers
    Daniel

  32. “Cairo is our failsafe implementation. It’s the backend we’ll use when we explicitly choose not to use hardware acceleration”

    I’m worried about how this will affect implementation of bug 486918.

    Many people have different reasons for not using hardware acceleration. To some people, the graphics card is a valuable resource better used elsewhere for various OpenCL computing, etc.

    Are you going to doom these people to forever have the worst resize filtering among all browsers? (See bug 667850, Cairo can’t even match bilinear…)

  33. [...] front, the huge 20% speed-up is due to the addition of a new graphics backend in Firefox 7 called Azure, which is a unified 2D graphics API that Firefox can use across every platform. At the moment, [...]

  34. [...] 7 and 8 run a new graphics engine called Azure, which you can read more about here.  And, in broad benchmark tests, ExtremeTech reports the following results: "Firefox 8, which [...]

  35. [...] front, the huge 20% speed-up is due to the addition of a new graphics backend in Firefox 7 called Azure, which is a unified 2D graphics API that Firefox can use across every platform. At the moment, [...]

  36. [...] fronte Canvas 2D, l’aumento di performance del 20% è dato dall’introduzione in Firefox 7 del progetto Azure, delle API grafiche 2D unificate che Firefox può sfruttare su varie piattaforme. Azure dovrebbe [...]

  37. This all sounds brilliant!

    So, is Azure going to be feature complete and hence be a viable replacement for the Cairo engine in say the Linux environments?

    That would be even more exciting!!!

  38. [...] front, the huge 20% speed-up is due to the addition of a new graphics backend in Firefox 7 called Azure, which is a unified 2D graphics API that Firefox can use across every platform. At the moment, [...]

  39. [...] Firefox 7, and matches Chrome 14. It’s due to the introduction of new graphics engine called Azure which can directly writes to Direct3D/OpenGL. The earlier Firefox using Cairo as its graphics [...]

  40. [...] performance. One of the ways they’ve done this is by adding a piece of software known as Azure. It’s Firefox’s new unified graphics API, and I’ve got to admit, they did a [...]

  41. [...] performance. One of the ways they’ve done this is by adding a piece of software known as Azure. It’s Firefox’s new unified graphics API, and I’ve got to admit, they did a [...]

  42. [...] aux problèmes que leur occasionnent Cairo, Mozilla a décidé de créer sa propre API dénommée : Azura. Si le nom peut prêter à confusion pour certains, cette nouvelle API est très intéressante et [...]

  43. [...] can find out all about Azure other blog posts, there’s an introduction from Joe Drew and there’s several more in detailed posts discussing the Direct2D Azure [...]

  44. [...] ability. With better 2D graphics, user will have more chances to experience high quality graphics. It’s going to be stateless, and significantly closer to Direct2D. We’re also going to develop new [...]

  45. [...] Un motor gráfico optimizado mediante el uso de la tecnología Azure. [...]

  46. [...] questo backend è utilizzabile solo con Windows 7 o Vista (altro sottoprogetto Mozilla: Azure, che usa ovviamente [...]

  47. [...] version est clairement annoncée  comme une transition vers Firefox  8, nom de code Azure. Ce projet a pour but de rendre l’affichage plus efficace des pages web et de pouvoir générer [...]

  48. [...] than before, which can, for example, make a difference when playing games in the browser. The technological basis for the improvement is the Azure API, which works with Direct2D under Windows and is expected to be [...]

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>