The First Third-party Test Pilot Study!


Test Pilot is a platform for studying Mozilla internal products, but more importantly it is an opportunity for the web community as a whole to evaluate ideas. It allows non-Mozilla projects to develop their own studies that aim to benefit the community and improve the Web.

Recently, we (the Web Security group at Carnegie Mellon University) released the first third-party Test Pilot study – Evaluation of Proposed Security Standard – in hopes of evaluating the compatibility of Entry Isolation, a newly-proposed browser security policy. The result of our study could help browser vendors and website developers decide whether this new policy is suitable for adoption.

Our study

The study we released on Test Pilot is a part of our ongoing research on securing websites and applications from malicious web entities. The security policy we are proposing allows web applications to specify a list of URLs that third-party sites can link to. When a third-party website requests a sensitive URL that is not on the list, the request will be blocked.

This whitelist-based security policy relies on participating websites having distinctive landing pages, which makes it impractical for socially integrated sites (e.g., Facebook). As such, we want to find out exactly what type of websites may benefit from this security policy, and its practicality.

In our Test Pilot study, we simulated a prototype of our security policy on ten websites. For each site, we whitelisted only the homepage, and recorded every time we ran into an incompatibility issue. A more detailed description can be found here.

This study ran for three days, and gathered data from more than 230,000 unique users. We were extremely grateful to the many users who participated, and Test Pilot for enabling us to collect such a large data set.  The sheer amount of data would have been practically impossible to collect with any other test platform.

Our Results

Initially, we believed security-sensitive sites such as Gmail or online banking sites would have an easy time adopting our Entry Isolation security policy. We did not expect many third-party sites to link deeply into these sensitive sites. Contrary to our beliefs, we found that security-sensitive sites get linked almost as frequently as other sites (e.g. CNN and New York Times).  Most of these third-party links differed from our selected landing pages, triggering violations in our policy.

We believe that for security-sensitive sites, most links are simply to different landing pages, rather than to content deep in the website like for other websites. As such, we plan to release a new version of our study soon, adding alternate landing pages into the policy in hopes of mitigating these compatibility issues.


Since our study was the first third-party study released by Test Pilot, we have been working very closely with the Test Pilot team to make sure we provide a pleasant experience for all users who join our studies. Unfortunately, due to our lack of experience with the platform, we may have missed a few bugs and caused inconvenience for some users. For this, we offer our sincere apology. Test Pilot is maturing steadily, and we plan to improve our study release process to ensure the final version will pass all necessary tests and reviews. We are open to suggestions if anyone wishes to help.

We would like to highlight again that both our Web Security group and the Test Pilot team will always abide by the following principles:

  • Respect user privacy: no personally identifiable information will be collected.
  • Give users full control of their study: they have the freedom to cancel the study at any time, or choose not to submit the data.

Get involved

Here are some ways that you can get involved with our study or Test Pilot in general:

  1. Help review the studies before they get released.
  2. Learn more about Test Pilot
  3. Develop your own studies.