14
Oct 11

Results of the inaugural BugKill day

Last Friday, we had the inaugural BugKill day, trying to clean up our graphics bugs. During this day, all 10 graphics developers, and 2 lovely people (Scott Johnson and Chris Lord) who aren’t nominally part of graphics, spent their whole day going through all of our open bugs, trying to reproduce them and/or determine if they’re still applicable.

We were able to close 308 bugs out of the 493 total we touched, which translates to a pretty fantastic 63% close rate. (There were bugs we didn’t touch but did look at during this triage, so this percentage is a maximum, but it’s still quite good.)

To do this triage, I ran a Bugzilla query for all all open bugs in the components GFX: Color Management, Graphics, Imagelib, Canvas 2D, and WebGL. (This was about 2500 bugs.) To divide this among our participants, I divided this long list by adding a regex query parameter on Bug ID to choose bugs by last number. (For example, [1]$ for all bugs ending in 1.)

Some of our observations:
* Having an open line to people (e.g. a Skype or Vidyo call) makes triaging easier. There are lots of bugs that are essentially “do such-and-such code cleanup”, and it’s difficult for someone who’s otherwise inexperienced in a piece of code to know if those bugs still apply.
* Being able to shuffle bugs to others’ queues would have been handy. While it’s useful to get fresh eyes on a bug, sometimes bugs will be much more efficiently handled by a domain expert. More on this a little later.
* There were several bugs with patches attached, but no review requests, and nobody had touched them for 4+ years. Bitrot was significant. It’d be worthwhile to do a quick query for patches in your components that haven’t got any review requests; you might be surprised.
* There were other bugs that we’d forgotten for no good reason, and which had easy patches written while triaging. (!)
* Being bold is better than asking people. Resolve bugs as INCOMPLETE, with the invitation to reopen, if you can’t reproduce and need more information from the reporter.

We still have >2100 bugs left open, and if our BugKill experience is any indication, a majority of them aren’t applicable anymore. In the mean time, we’re going to continue doing off-and-on bug killing on the list of bugs we were assigned. And to add to that ad-hoc triage, we’re going to do another graphics BugKill on Friday, November 4, 2011: 3 weeks from today.

However, we’re going to make one important modification to our process: we’re going to allow people to send bugs to others’ lists by (ab)using the QA Contact field. That is, everyone’s queries will have a list of bugs selected by regex, and will also include every bug where the person is the QA Contact. People can send bugs to others by changing QA Contact, and once that person’s triaged that bug, they’ll reset the QA Contact. I encourage others who are considering BugKill to do this from the start.


03
Oct 11

Graphics BugKill day: Friday, October 7, 2011

On Friday, October 7, the Graphics team is going to be having the inaugural BugKill day.

Currently, there are over 2000 bugs in various graphics-related bugzilla components. This is too many for any person to keep in her head.

Our goal is to get down to about 300 bugs per component by a mix of closing obsolete bugs and creating new components. We’ll then be in a position to keep our bugs in check, because each component will have one or two people able to monitor and triage bugs.

During this BugKill day, everyone involved will be given a Bugzilla query, and they’ll be responsible for going through, bug by bug, and determining whether the bug still applies. Due to the volume of bugs, we don’t expect to be able to get through them all, but we should be able to process several hundred.

If you’re interested in participating as a BugKiller, please let me know before the end of the day next Tuesday, October 4th so I can divide up the bugs. You’ll be joining the entire graphics team, who will be spending their full working day BugKilling.

If you’re interested in observing, we’ll be working through our bugs on irc.mozilla.org in channel #gfx, and we’ll also likely have an open video channel for discussion (the Graphics Vidyo room).

I’ll also have a post-BugKill report to talk about common issues, pitfalls, how successful we were, and where we’re going to go from our current state.


26
Apr 11

Introducing the Azure project

Motivation

Firefox 4′s graphics performance is great, and some last-minute “low-hanging fruit” inefficiencies Robert O’Callahan found made us even faster on some demos in Firefox 5. We’re not content with “great”, though, and our investigations into how to make drawing even faster have revealed that some of our choices in Gecko’s graphics engine aren’t optimal for performance.

How Mozilla renders

To display a web page, Gecko’s layout engine looks at how that page is constructed, and divides it into independent layers. Each of those layers is drawn using Cairo, a 2D graphics API built on top of the operating system’s graphics APIs. Cairo’s drawing into layers can be hardware accelerated — this is known as “content acceleration” — or done in software. Right now, content acceleration is fastest on Windows Vista and 7, where we use Direct2D, but we want to be able to extend good acceleration to all operating systems, including mobile.

After drawing, layers are composited together using either the platform’s 3D APIs (OpenGL or Direct3D) or Cairo, and the result is displayed on the screen.

Format conversions

Cairo uses a fixed point representation of numbers. Unfortunately, this means that when Cairo calls get mapped to an API like Direct2D, which uses floating point numbers, two conversions are required: one from from your internal floating point representation to Cairo’s fixed point representation, in which it is initially stored, and then back to floating point again when Cairo makes Direct2D calls.

Statefulness

Cairo is a stateful API. This means that any given function call depends on the results of the previous function calls to determine its result. Unfortunately, statefulness strictly adds overhead to an intermediate graphics API: if it maps to another stateful API, Cairo has to track its own state separately from the underlying API (usually due to subtle differences between the two). Also, Cairo uses a stateless surface API internally; whether the underlying API is stateful or stateless, there’s always a conversion from one to the other. Further, the HTML5 Canvas element is a stateful API that’s built on top of Cairo, and it has to have its own state tracking too.

What makes this worse is that Gecko is better-suited by a stateless API; when we begin rendering any part of a web page, we need to reset most of Cairo’s stored attributes, since they’re determined by the element’s CSS.

What can we do about it?

We would very much like to remove these unnecessary conversions and state tracking. We’d also like to be able to hardware accelerate content drawing in an easy and cross-platform way.

Mozilla needs a stateless graphics API that’s closer both to platform APIs and hardware.

This API needs to be close enough to Direct2D to be little more than a thin wrapper, but general enough to be implemented on OpenGL and Direct3D 9.

The Azure project

To accomplish this, the Mozilla graphics team is creating a new 2D graphics API. It’s going to be stateless, and significantly closer to Direct2D. We’re also going to develop new “3D” backends for drawing content in an accelerated way to OpenGL, Direct3D 9, and Direct3D 10.

The backends

Once complete, the new 2D API will be implemented by several different backends, depending on OS, hardware, and driver. We will also have a fallback implementation, using Cairo, that we’ll use when we don’t have a more specialized backend available.

Direct2D

We don’t make the best use of Direct2D right now. Part of it is the floating/fixed point impedance mismatch I described earlier, but part of it is capabilities Cairo exposes that Direct2D doesn’t implement. These need to be worked around or emulated using software. The API we create as part of the Azure project will be close enough to Direct2D to not require fallbacks.

Quartz / CoreGraphics

On OS X, Cairo uses its Quartz backend for all drawing. Quartz can be fast: Safari is faster than Firefox on some demos, even though we both use the same backend for drawing. We believe one reason this is the case is because Quartz is stateful: Cairo needs to convert from its stateful API to its internal stateless surface API, then back to Quartz’s stateful API. We’ve designed Azure’s stateless 2D API so that it can map from stateless to stateful in a more optimal, and optimizable, way.

Cairo

Cairo is our failsafe implementation. It’s the backend we’ll use before others are ready or when we explicitly choose not to use hardware acceleration, for example because of driver issues. We also need Cairo for printing support.

It’s possible that we will decide to modify our internal copy of Cairo to allow us to use the internal stateless surface API directly, since we think Cairo’s surface API is a pretty good match to Azure’s 2D API.

3D Backends

The goal of the “3D” backends is to implement Azure’s 2D API on top of the common 3D APIs (OpenGL and Direct3D) so we can have content acceleration on Windows XP, OS X, Linux, and mobile. Our intent with these 3D backends is to be fully cross-platform, including Windows 7. This lets us better control our performance characteristics and bugs, rather than spending a lot of time and energy working around Direct2D bugs.

Electrolysis and Azure

The Azure 2D API will make it possible for us to hardware accelerate content drawing, even in the distant future when we “sandbox” our content processes. This new 2D API is required for hardware accelerated sandboxed content processes, but isn’t required for the initial Electrolysis roll-out.

Timeline and checkpoints

Over the past few weeks, we created a proof-of-concept Direct2D backend to test our hypothesis that we could create a faster non-Cairo API. Our hypothesis was correct — our backend is significantly faster than the Direct2D Cairo backend on some commonly-used Canvas demos and benchmarks. This work is all available in the graphics project branch.

Our initial success led to our current plan: to design and test this new 2D API as a backend for the HTML5 canvas element. The graphics team has included getting the Azure Direct2D canvas backend enabled in mozilla-central as a Q2 goal.

We are also in the middle of writing similar canvas backends for Cairo and Quartz, both of which are already in the graphics project branch, but those are not yet operational. We haven’t defined specific time goals for those backends.

Postscript: Why “Azure”?

Jim Clark invented the first GPU at Stanford in 1979, called the “Geometry Engine”. He founded SGI based on this work.

SGI made many workstations and servers with “colour” names like Onyx, Indigo, and Crimson. They are also the original inventors of OpenGL.

After SGI, Clark founded a little-known software company called Netscape.

It seemed only fitting to pay homage to him in this small way!


23
Nov 10

Why is HWACCEL slow on my computer?

There’s a pretty impressive demo called HWACCEL which is quite slow on computers that are ostensibly hardware accelerated. To understand why, you need to understand a little bit about how HWACCEL is built.

A little bit about HWACCEL

HWACCEL creates a big HTML5 canvas, then uses the canvas drawImage function to draw lots of images.

Why is canvas slow on hardware accelerated computers?

Canvas is a general drawing API. Hardware accelerating general drawing APIs is quite hard; Microsoft wrote a framework called Direct2D that uses your DirectX 10 hardware to hardware accelerate a lot of different things, and we use this framework where it’s available (Windows Vista and 7). It doesn’t exist anywhere else, though, so we can only use the somewhat more limited 3D functionality provided by frameworks like Direct3D and OpenGL on Windows XP and Mac OS X. (For more information on the difference, see my previous blog post on how we are going to hardware accelerate Firefox 4.)

So canvas is always going to be slow, then?

Not if we can help it! Writing a hardware accelerated backend to canvas is tough, but it’s doable. We might try to use Cairo’s OpenGL backend. We also might try collaborating with some of the other browser maker. As I’m sure it’s clear, we haven’t quite worked out what the path to success will be, just that we need one.

When’s all this going to happen?

Later. Not for Firefox 4, for sure; with some hard work and a little bit of luck, we might have something for 4.next. We welcome folks helping us out – right now, that means helping us fix the Graphics bugs blocking us from releasing Firefox 4. Join us on Mozilla IRC, channel #gfx, if you’re interested!


10
Nov 10

Hardware Acceleration in the latest Firefox 4 beta

The most recent Firefox 4 beta has enhanced support for hardware acceleration on Windows and Mac OS X. This release of Firefox will be faster on more webpages while at the same time being compatible with more computers.

Direct2D improvements

On Windows Vista and 7, Firefox hardware accelerates web content using Microsoft’s Direct2D framework. The latest beta of Firefox fixes bugs that made Firefox stop drawing when the graphics driver restarted. It also now has a graphics “block list” that makes sure Firefox doesn’t try to use graphics drivers or chips known to cause problems.

Hardware accelerated compositing

Firefox now hardware accelerates the composition of layers on Windows XP, Vista, and 7, and Mac OS X. If you’re running Windows, Firefox uses Direct3D-accelerated layers, while on Mac OS X, it uses OpenGL for the same task. Hardware accelerating layer composition will make some sites significantly faster; you will notice faster scrolling and faster interaction with graphically-intensive web pages. Firefox also accelerates the display of HTML5 video, giving a special boost to pages that use it.

Testing hardware acceleration

You can help us develop Firefox 4! Just use the Firefox 4 beta as your everyday browser and tell us about your experience via the Firefox Feedback button. If you notice something specific has gone wrong, you can also file a bug in Bugzilla; we have information on how to gather information for your hardware acceleration bug report too.


10
Nov 10

How to tell if you’re using hardware acceleration

Sometimes, Firefox 4 can’t use hardware acceleration. Usually, this happens because Firefox has identified your graphics card (GPU) or driver (the software that runs the GPU) as being unreliable.

To tell if you’re using hardware acceleration in Firefox, enter the address about:support into your address bar. At the bottom of that page is a section labeled “Graphics.” If you’re using hardware acceleration, this section will contain a message similar to “GPU Accelerated Windows: 1/1 Direct3D 10″.

about:support graphics section with hardware acceleration enabled

Hardware Acceleration enabled

However, if hardware acceleration is disabled automatically by Firefox, the about:support Graphics section will simply display the message “GPU Accelerated Windows: 0/1.” (Frequently, issues with graphics drivers are resolved by updating those drivers. Contact your computer or graphics card manufacturer for more details.)

about:support Graphics section with hw accel disabled

Hardware Acceleration disabled


15
Sep 10

Testing hardware acceleration

Firefox 4 nightly builds (and soon, betas) now include hardware acceleration on all versions of Windows! We would simply love it if you would help us test this acceleration.

To know if you’re using hardware acceleration, check the Graphics section of about:support. If your computer is capable of using content acceleration using Direct2D, you will see “Direct2D Enabled: true.” If your computer is capable of using layers/compositing acceleration using Direct3D 9, you will see “GPU Accelerated Windows: 2/2 Direct3D 9″[1] (for example).

To test if a problem you’re seeing is due to hardware acceleration, you can disable it. There are two ways to accomplish this:

  • Run Firefox in safe mode. (All hardware acceleration is always disabled in safe mode.)
  • Uncheck “Use hardware acceleration when available” in the Advanced section of the Preferences/Options dialog.

When reporting a bug caused by hardware acceleration, please copy the entire Graphics information block from about:support into your bug report. This will help us work out root causes and reproducing your problems.

In the very near future, we plan to enable layers/compositing hardware acceleration on Mac OS X using OpenGL. The same testing procedure works there, too, and we will even more gratefully accept your bug reports for hardware acceleration issues on OS X.

1. Due to some technical issues, each window can have a different hardware acceleration status, which is why we show a ratio. Most of the time all of your windows will be accelerated, though.


25
May 10

Hardware accelerating Firefox

Bas Schouten has posted a lot about hardware accelerating Firefox, specifically with Direct2D and OpenGL. However, there’s been some confusion as to what it is we’re going to hardware accelerate, and how we’re going to do it. This blog post aims to be the definitive reference for hardware acceleration in Firefox.

Please note: We have committed to turning on certain parts of hardware acceleration in developer previews, but that’s not a guarantee of those bits shipping in Firefox 4. We’re going to try hard to ship some form of acceleration, though.

Layers

Layers is a technology that lets conceptually simple, but computationally expensive, parts of the web get offloaded to GPUs. Examples of this include transparency, scaling, composition, and simple animations of those attributes. Since we’re dealing mostly with images on the GPU, we can also accelerate operations on these images, like converting colours; this is especially useful in the case of video, because we can get the GPU to convert from video’s native YCbCr to the GPU’s native RGB, and then scale the result, for example to fullscreen. Both of these operations are expensive on the CPU, but relatively free on the GPU.

The downside of layers is that, in order to get the sort of benefit that’s possible from GPU acceleration, a lot of analysis of the document needs to be done to identify the parts that need to be separated into their own layers. Some of this is relatively easy, like background images; other parts are much harder. Also, layers is designed to accelerate only portions of the web; this means that, in the common case, we will render most of the web page using software, then do only the hardest/slowest part using the GPU directly.

Code currently in mozilla-central supports layers in three modes: a basic software-only mode (using Cairo); a mode that combines software with OpenGL; and a mode that combines software with Direct3D 9 (D3D9). At the time of this writing, only full-screen HTML5 video is rendered using our GPU accelerated (OpenGL and D3D9) layers backends, but in the future we plan to accelerate all of our rendering. To test this hardware acceleration, right-click on a video and select “Full Screen.” GPU accelerated full-screen video is turned on by default; you can check its status in your error console.

Direct2D

Direct2D (D2D) is a Cairo backend that GPU accelerates everything displayed by Firefox. Some parts of the web are more amenable to acceleration than others; for example, moving and scaling images around (like in photos.svg) is much faster. The average page that displays a lot of text and perhaps a few images won’t see nearly as big a speedup, though. The Direct2D Cairo backend doesn’t require any special analysis like the layers backends require, because everything in Firefox is already rendered using Cairo.

The downside to Direct2D is that it’s only supported on Windows Vista + Platform Update and Windows 7, and only then on relatively new GPUs (generally, GPUs that support DirectX 10).

Direct2D support is currently turned off by default, but can be turned on manually using about:config.