Testing WebRTC on Android

Testing WebRTC is challenging. It is inherently timeout based. Tests that work well locally often fail when run in automation. At Mozilla, we run the majority of our Android tests on ARM emulators on AWS instances. In case anyone is wondering, this is very slow. We’ve recently started running our tests on actual phones as well as on emulators. Running faster should lead to fewer performance related intermittent failures.

We first looked at running tests on x86 emulators. Unfortunately, we’ve had only limited success in the past getting our tests to run stably on x86, even on a local emulator. Beyond this, the hardware virtualization support required to run x86 emulators truly quickly is not available on AWS, so we’d see only limited benefits here.

Our WebRTC tests were always on the edge of timing related failures. This caused problems with intermittent failures. Even worse, seemingly innocuous changes could push things over the edge, leading to a lot of test failures and delays landing important patches. These failures would more or less never reproduce when run locally. Because of this we ended up disabling more and more tests on Android.

It is surprisingly difficult to run tests on mobile devices in a repeatable fashion. We were lucky that Mozilla already had Autophone available. Originally developed to run performance tests on actual phones, it is a nice fit for the sort of real time constraints we see with WebRTC testing.

We added functionality to Autophone to only run tests if the commit affected WebRTC related code. This allows us to keep up with the large volume of commits each day. Unfortunately, if seemingly unrelated code were to cause something to break, we’d be stuck backfilling jobs in order to be able to bisect the failure. So far, this hasn’t been a problem.

The new jobs were recently enabled on mozilla-inbound and autoland. So far, things are looking good. We’re still running tests on emulators, but as mentioned above, we had to disable a lot of them.

Running the Tests

The new jobs appear on Treeherder as Mw and Cw. The mochitests are run locally using:

./mach mochitest --manifest obj-arm-linux-androideabi/_tests/testing/mochitest/manifests/autophone-webrtc.ini dom/media/tests/mochitest/

and are available on Try using the following syntax:

try: -b do -p android-api-15 -u autophone-mochitest-webrtc -t none

The same manifest is used to run crashtests on Autophone as on the emulators, so no special mach command is necessary. To run the crashtests on Try, use autophone-crashtest-webrtc instead of (or in addition to) autophone-mochitest-webrtc.

Because the mochitests use a different manifest file for Autophone, you need to edit

testing/mochitest/manifests/autophone-webrtc.ini

to enable and disable tests rather than the usual mochitest.ini file.