xpcshell manifests, phase 2

June 29th, 2011

Recently we implemented a manifest format for xpcshell unit tests (with Joel Maher doing the lion’s share of the work). After that work landed, we realized there were some things missing from our initial design, so we set out to revamp a few things to make it easier to write useful manifests.

We decided to make the manifests support boolean expressions, similar to what reftest manifests allow, except with a restricted grammar, and not “all of JavaScript”. To make this useful, we had to offer a set of values to test, so Jeff Hammel buckled down and wrote a Python module called mozinfo that had been in discussion for a long time. I wrote a few bits to hook this all up between the build system and the xpcshell test harness, and it all landed on mozilla-central this morning.

I’ll update the MDN documentation later today, but for a preview, a sample manifest entry might look like:

[test_foo.js]
skip-if = os == 'win' || os == 'linux'

If you look in your object directory in a build containing these patches (or in the xpcshell directory of a test package from a tinderbox build), you’ll find a mozinfo.json, which is where most of the values you can use in these expressions come from. For example, the mozinfo.json for my 64-bit Linux build looks like:

{'os': 'linux', 'toolkit': 'gtk2', 'crashreporter': false, 'debug': false, 'bits': 64, 'processor': 'x86_64'}

You can annotate tests with “skip-if, run-if and fail-if” conditions currently. “skip-if” indicates that a test should not be run if the condition evaluates to true, “run-if” indicates that a test should only be run if the condition evaluates to true and “fail-if” indicates that the test is known to fail if the condition is true. Tests marked fail-if will produce TEST-KNOWN-FAIL output if they fail, and TEST-UNEXPECTED-PASS (which is treated as a failure) if they pass.

Hopefully this work will enable developers to more easily work with xpcshell tests. We’d appreciate any feedback you have on these changes!

For a long time when our unit tests or Talos performance tests encountered a crash, the result was nothing but frustration. If you were lucky, you could tell that it crashed, but you had no idea where. Poor Blake spent weeks tracing down a crash from his speculative-parsing patch that only seemed to occur on Talos. Up until recently I figured the only way to make this happen was going to involve a fair amount of work that only I was going to be able to do. A few weeks ago it was determined that this was becoming a significant impact on development, as patches would get checked in, cause a crash and be backed out, leaving the developer with nothing to go on.

Benjamin Smedberg has been hard at work making it possible to get stacks in this situation, using the same Breakpad utilities we use on our Socorro server, but locally on the machine running the tests. Practically all of the pieces were in place this afternoon when #developers cornered Alice and closed the tree while she landed the final patch to make Talos produce stack traces. Boris then committed a test crash, and as a result we were able to see crash stacks in Mochitest (OS X, Linux) as well as Talos (OS X, Linux).

Thanks to Benjamin for doing most of the heavy lifting here, and for
Alice for taking the Talos part across the finish line. The Talos work
was mostly in bug 480577, and the unit test work was bug 481732. Note
that currently this only works in Mochitest (all 4 varieties), it will
work in Reftest/Crashtest after bug 479225 is fixed (which should be soon).

(Cross posted in dev.tree-management, but posting here for a wider audience.)

Unit tests: now with less suck!

November 21st, 2007

Thanks to the combined efforts of a few people, the Tinderbox build logs for our unit test machines now suck much less.  You can now click on “View Brief Log” and get a summary of test failures right at the top, instead of searching through the full log for various failure strings.  In addition, if you click down to the errors in the body of the log, the test files are linkified to bonsai for you.  Awesome!

RLk:0B (and staying that way)

October 24th, 2007

So, some time ago dbaron got RLk down to 0 bytes on our leak test box.  Sometime after that, we deployed the new Linux reference platform, only to have that go back up to 8 bytes.  Turns out it was my fault, a string wasn’t being freed in the crash reporter code.  The crash reporter must not have been enabled on the previous reference platform.  I’ve made amends and fixed this, and I also checked in rhelmer’s patch to make the leak test boxes turn orange if RLk goes above zero, so we should be able to hold the line on this per our test failure policy.  For comparison, on the 1.8 branch we leak up to 45KB(!) per test run.