A New Firefox Add-ons Validator

The state of add-ons has changed a lot over the past five years, with Jetpack add-ons rising in popularity and Web Extensions on the horizon. Our validation process hasn’t changed as much as the ecosystem it validates, so today Mozilla is announcing we’re building a new Add-ons Validator, written in JS and available for testing today! We started this project only a few months ago and it’s still not production-ready, but we’d love your feedback on it.

Why the Add-ons Validator is Important

Add-ons are a huge part of why people use Firefox. There are currently over 22,000 available, and with work underway to allow Web Extensions in Firefox, it will become easier than ever to develop and update them.

All add-ons listed on addons.mozilla.org (AMO) are required to pass a review by Mozilla’s add-on review team, and the first step in this process is automated validation using the Add-ons Validator.

The validator alerts reviewers to deprecated API usage, errors, and bad practices. Since add-ons can contain a lot of code, the alerts can help developers pinpoint the bits of code that might make your browser buggy or slow, among other problems. It also helps detect insecure add-on code. It helps keep your browsing fast and safe.

Our current validator is a bit old, and because it’s written in Python with JavaScript dependencies, our old validator is difficult for add-on developers to install themselves. This means add-on developers often don’t know about validation errors until they submit their add-on for review.

This wastes time, introducing a feedback cycle that could have been avoided if the add-on developer could have just run addons-validator myAddon.xpi before they uploaded their add-on. If developers could easily check their add-ons for errors locally, getting their add-ons in front of millions of users is that much faster.

And now they can!

The new Add-ons Validator, in JS

I’m not a fan of massive rewrites, but in this case it really helps. Add-on developers are JavaScript coders and nearly everyone involved in web development these days uses Node.js. That’s why we’ve written the new validator in JavaScript and published it on npm, which you can install right now.

We also took this opportunity to review all the rules the old add-on validator defined, and removed a lot of outdated ones. Some of these hadn’t been seen on AMO for years. This allowed us to cut down on code footprint and make a faster, leaner, and easier-to-work-with validator for the future.

Speaking of which…

What’s next?

The new validator is not production-quality code yet and there are rules that we haven’t implemented yet, but we’re looking to finish it by the first half of next year.

We’re still porting over relevant rules from the old validator. Our three objectives are:

  1. Porting old rules (discarding outdated ones where necessary)
  2. Adding support for Web Extensions
  3. Getting the new validator running in production

We’re looking for help with those first two objectives, so if you’d like to help us make our slightly ambitious full-project-rewrite-deadline, you can…

Get Involved!

If you’re an add-on developer, JavaScript programmer, or both: we’d love your help! Our code and issue tracker are on GitHub at github.com/mozilla/addons-validator. We keep a healthy backlog of issues available, so you can help us add rules, review code, or test things out there. We also have a good first bug label if you’re new to add-ons but want to contribute!

If you’d like to try the next-generation add-ons validator, you can install it with npm: npm install addons-validator. Run your add-ons against it and let us know what you think. We’d love your feedback as GitHub issues, or emails on the add-on developer mailing list.

And if you’re an add-on developer who wishes the validator did something it currently doesn’t, please let us know!

We’re really excited about the future of add-ons at Mozilla; we hope this new validator will help people write better add-ons. It should make writing add-ons faster, help reviewers get through add-on approvals faster, and ultimately result in more awesome add-ons available for all Firefox users.

Happy hacking!

8 comments on “A New Firefox Add-ons Validator”

  1. Ben Bucksch wrote on

    Thank you for re-writing the validator. It’s badly needed!

    It’s particularly urgent to re-write the rules as well. Many of them are simple string-based patterns and fall ridiculously short in being helpful. Our ext causes endless lines of warnings, most of them false positives. The rules don’t distinguish between good and bad code, but mark anything that *could* be bad.

    The problem arises when:
    * There’s a (human) reviewer that doesn’t read or understand the code carefully.
    * These rules are used for denying automatic signing.
    In combination with the “stop when in doubt” rule, and the fact that the rules are currently so stupid that they raise a lot of false positive falgs, that means that the majority of exts cannot be automatically signed.

    Particularly that automatic signing was *promised* as the solution so that self-hosted ext can continue to operate as they did. If I have several days (or even several weeks!) delay between my fix and QA and distribution, and I can’t even get my own fixes to my own users via my own servers in a time manner, that’s a massive practical problem and qualifies as censorship. That’s why it’s critically important that ext can be automatically signed.

    And that’s why it’s critically important that the rules flag ONLY those code parts that are DEFINITELY bad.

    Example: We used the newtab change observer to know when we should turn off (!) our newtab feature, because another ext grabbed it. However, we got automatically flagged and blamed for engaging in a grab war. Turned out, we did the exact opposite, but of course also with a listener. The check triggered on the listener, and the reviewer didn’t notice that our code is fine (and neither did I).
    The same is true for an endless list of other checks.

    OTOH, the current checks can trivially be circumvented with a var prefname “n” + “ewt” + “ab.u” + “rl” or similar, so they don’t actually catch the bad guys.

    A simple run-time check would have shown that we don’t do anything bad. I.e. execute the ext (in a sandbox) and watch its actions. That allows you to catch a lot more bad guys that try to evade the checks, and get less good guys in your net.

    So, I BEG you, please please, do not re-write the rules as they were. Make them smart and precise and correct. The old ones were an ocean far from that. This change need to start at the design phase, so I think the time for change is now, with exactly your project. Please do not perpetuate the old bad design choices. I BEG you, please design it to be precise from the start.

    1. Matthew Riley MacPherson wrote on

      Circumvention of checks using malicious string assembly is virtually impossible to detect without a massive engineering effort. I looked into what it would mean to catch things like this early on, and the (short) answer was “we can’t”. Because of the nature of JS and the ease of that kind of obfuscation, we need to rely on manual reviews for that kind of thing, sadly.

      Doing run-time checks could be explored in the future, but it requires a lot of infrastructure and right now we’re looking for an add-on validator that’s easier to run, not harder 🙂

      For me, the validator isn’t the most aptly named and should be thought of more as a linter than a 100% validator.

  2. Charles wrote on

    “developers often don’t know about validation errors until they submit their add-on for review”

    Ugh, it seems that the next has been really forgotten:

    …that let to run the add-ons validator against any xpi without be it part of the process of submiting/updating an add-on. It has been available for years but the changes done to it in the last months seems to assume that it is always run as part of the review process, so I’m not sure if it still works correctly. Please check it.

  3. Brett Zamir wrote on

    Excellent to see the use of JavaScript in such projects! A worthy effort!

    If you really want to dogfood things all the way, maybe you could avoid even Node.js by implementing as an add-on (with a custom command line listener).

  4. Wladimir Palant wrote on

    Tried it, got: “Multiple elements found.” Sure enough, the validator looks for everywhere, not merely at the top level. That particular issue was fixed in the repository already (https://github.com/mozilla/addons-validator/commit/0f0ee973ac6dab7840a42f25b817ed60fdab32a3) but I wonder whether it will really work in all scenarios – that’s RDF, not your regular XML document, there are lots of different ways to write the same thing.

    That’s the crux with rewrites, they tend to reintroduce bugs that the developers of the original code fixed long ago (and: yes, I remember them having their share of RDF-induced problems as well but that was years ago).

  5. Wladimir Palant wrote on

    Heh, it should have been “Multiple <em:name> elements found.” above, also “validator looks for <em:name> everywhere”

  6. Mindaugas wrote on

    Why the name change?

    1. Matthew Riley MacPherson wrote on

      We decided it really was more of a linter than a validator, and even replied to a few folks agreeing with that idea. We talked about it in meetings and on GitHub a bit and decided it gave users a better expectation of what the “validator” could do and what it couldn’t. For instance, it’s hard for us to deal with obfuscated code because we don’t actually run it (we could, in a sandbox, but that would be another project’s worth of work ).

      So we renamed it to match expectations of a linter, not a validator.