There has been a lot of activity since the previous report, mostly along three lines of development: multithreaded profiling, getting profiling working for Android nightlies, and improving the quality of the stack traces.

We now have native unwinds working out-of-the-box for 32- and 64-bit Linux nightlies, and very nearly working for Android nightlies.

Multithreaded profiling:

• Benoit landed a big change (734691) to the profiler backend, that makes it able to profile multiple threads.  Work is in progress (861863) to update the Cleopatra GUI to match, but it has not yet landed.  Early adopters are welcome to try out this new functionality.

• Benoit also landed 788022, which adds support for profiling Java code running on the Dalvik VM on Android.

Getting profiling working for Android nightlies:

• On Android, we landed plumbing code that allows Breakpad to read debug info direct out of APK files, by interfacing with faulty.lib (802240, 861141).

• On Android, we landed a 1-liner that enabled profiling on nightlies (863264, 863375).  Then the fun started.  A few reftests started failing, so the patch was backed out.  After quite some digging, it turns out that this change had the effect of changing the compile flags from “-O2” to “-O2 -fno-omit-frame-pointer”, and gcc-4.6.2 wound up miscompiling gfxFont::RunMetrics::RunMetrics() in such a way as to copy 16 bytes of uninitialised stack-allocated garbage to the start of the object it is initialising.  Nathan Froyd suggested and landed a workaround.

Improving the quality of native stack traces:

• I made a simple change that periodically shows counts of how many frames were recovered by CFI data, how many by following frame pointers, and how many by stack scanning (863705).  Using this, it is finally possible to get some idea of how well Breakpad is doing and why it sometimes produces results that are poorer than we expect.

• Using the 863705 patch, I investigated unwinding behaviour on both 32- and 64-bit Linux, and concluded that we are doing well there.  In particular, Breakpad is able to unwind using frame pointers on 32-bit Linux, which had up to that point been somewhat in doubt. Any incomplete stack traces on that platform are due to system libraries which have been compiled without frame pointers and for which there is no CFI available.  There’s nothing we can do about them, except for resorting to stack scanning.  As discussed in the previous posting, stack scanning gives poor results and is disabled by default.

• Also on the theme of diagnosing unwind problenms, I modified SPS so as to produce CFI coverage statistics (859775).  This prints extra information at debug info load time, indicating how much CFI was read and how much address range it covers.  I had hoped to also be able to get the size of the relevant .text segment, so as to enable printing messages of the form “available CFI covers 85.7% of the text segment”.  This unfortunately didn’t work out due to the difficulties of getting the segment size.

• The idea of enhancing Breakpad to use EXIDX unwind info on ARM came back to life and was generally well received.  TedM refreshed his patch (863475) and it is now waiting for further resync with our local Breakpad changes.

Misc other fixes:

• A problem causing Firefox to livelock when starting any external program while the profiler is running (837390) was fixed.

• As it stands, SPS/breakpad will unwind up to 1024 stack frames before stopping.  That can happen if the stack is corrupted.  In the worst case this can potentially waste a lot of unwinder time, so we installed a 256 frame limit (859745).