Mozilla Add-ons Community Blog

Add-on Verification Suite is on AMO

14 responses

RJ Walsh has just completed the first revision of the add-on verification suite we proposed a few months ago, now live on addons.mozilla.org. This suite gives add-on authors and editors an automated tool to examine add-ons for common problems, and recommends fixes where appropriate.

The goals for the project are:

  • Detect problems in add-ons, including issues with security, compatibility, and other areas. The author can receive alerts if problems are found.
  • Help editors by automating simple, repetitive tests so they can spend their time doing more interesting things
  • Maintain the level of excellence in public add-ons that Firefox users expect

Accessing the verification suite

If you are an add-on author log in to your developer control panel and choose “Versions and Files” for one of your add-ons. Find the version you want to analyze and click “No Test Results.” This takes you to the testing screen. We’ll make this process easier in a future update.

What you’ll see

Verification Summary
After a brief analysis, a summary is displayed at the top of the page with what tests were run and their results. If you see all green check marks, congratulations! Your add-on has passed the verification suite’s current set of tests.

Verification Details
If you see any warnings or errors scroll down the page to see a detailed report. Beside each message that is reported is a [help] link which points to a page of documentation explaining each test in more detail.

Things to remember

Keep in mind that many tests are written to catch a broad number of cases and may display false positives. Seeing warnings in the add-on analysis does not necessarily mean your add-on has problems – it’s simply raising a flag to double check that area. As usual, we are very open about exactly what we are testing here. This tool, just like AMO, is completely open source and can be browsed online.

Future plans

Over the next few months we’ll refine the tool to prevent false positives and add additional tests. If you can think of other tests that would be helpful for authors or editors please submit a patch or just let us know. We expect to have the tests running automatically when an add-on is uploaded by the end of the year.

If you have any questions, comments, or ideas for improvements please let us know, either as a comment on this blog or on IRC.

14 comments on “Add-on Verification Suite is on AMO”

  1. Wladimir Palant wrote on

    Ah, I am finally able to run those tests 🙂
    Lots of nice warnings. However, finding /eval/ in localizations (particularly in the middle of a word) isn’t really helpful. Maybe change that to /\beval\s*\(/ at least?

  2. Philip Chee wrote on

    Yeah lots of bogus warnings since in my XUL I have several element IDs that contain the “eval” string e.g.
    toolbar id=”ToolbarEval”

    Phil

  3. Robert Kaiser wrote on

    Heh, “eval” as part of words in dictionaries also produces security warnings. 🙂

    And I saw that you choke on missing install.js for SeaMonkey compatibility on dictionaries, which is nice for MySpell dictionaries, but for HunSpell it’s wrong, as we only support those in 2.x and even SeaMonkey doesn’t need/want install.js there. So, if minVersion for SeaMonkey is 2.0a1 or higher, please omit this check for install.js

  4. Michael Kaply wrote on

    eval is also a problem in that it matches XPath:

    var xpathResult = (node.ownerDocument || node).evaluate(xpathExpression, node, null, Components.interfaces.nsIDOMXPathResult.FIRST_ORDERED_NODE_TYPE, null);

  5. Wil Clouser wrote on

    Thanks for the feedback guys.

    I’ve filed https://bugzilla.mozilla.org/show_bug.cgi?id=509038 about the eval regex – yeah, it’s a little broad.

    And https://bugzilla.mozilla.org/show_bug.cgi?id=509039 about the install.js + SeaMonkey.

  6. Bob Davies wrote on

    This is fantastic, great job guys, thanks. 🙂

    I noticed /eval/ also matches for evalInSandbox().

  7. Ken Saunders wrote on

    Coolness.

    1. Would be awesome (probably impossible) to simulate running/testing an add-on on various OS’s. Many to most developers only use or have access to one OS.

    2. How about the option to add an AMO validation badge/button to an add-on’s page?

  8. Wil Clouser wrote on

    1. Would be awesome (probably impossible) to simulate running/testing an add-on on various OS’s. Many to most developers only use or have access to one OS.

    Yeah, that would be difficult. We’re looking into VMs for editors to be able to test add-ons on, and we’re looking at some other automated testing like memory leak stuff. Nothing jumps to mind on how to install in multiple OSs with this tool and then actually use the add-on to see if it works. There are just too many variables for each add-on to be able to test functionality.

    2. How about the option to add an AMO validation badge/button to an add-on’s page?

    That’s an interesting idea. We could even make it automatically show up on the add-on page if there are no warnings/errors. I’m a little concerned that false positives with the current setup will mean “oh well” but will cause some discontent if we started doing public promotions like that.

  9. Alfred Kayser wrote on

    Note that the verification suite generates lots of false hits on themes:
    Things like:
    chrome/littlefox.jar/browser/feedview.css (39) : Matched Pattern: “/-moz-binding:(?!\s*url\s*\(\s*[“‘]?chrome:\/\/.*\/content\/)/”
    and:
    chrome.manifest (1) : Matched Pattern: “/^(?!(#|skin |style ))/”
    are actually OK for themes.

    So, the suite needs to be more tuned to Themes as well.

    Other than that, this is a nice addition to the developers toolkit!

  10. Wil Clouser wrote on

    Thanks Alfred. I filed https://bugzilla.mozilla.org/show_bug.cgi?id=509427

  11. Goofy wrote on

    may I suggest a test to check for hardcoded user interface strings?
    Or more simply a detection of the absence of locale support, triggering something like “It seems your extension has no locale support. Though it is no big issue, it is recommended to have at least one en-US locale subfolder for your user interface strings”
    (and a link to appropriate documentation)

    TIA

  12. Alan wrote on

    Nice tool, but it complains about “launch”, even when I’m using the launch() method of Components.classes[“@mozilla.org/file/local;1”].createInstance(Components.interfaces.nsILocalFile).

  13. Eric Jung wrote on

    We need a “Verification” component in bugzilla under the addons.mozilla.org product. In the meantime, can we add a reg exp to check for the string:

    .*\.setCharPref(\”.*\@site\.update\.url\”.* (reg exp untested)

    This will match, for example:

    prefs.setCharPref(“extensions.sample@site.update.url”, “http://badsite.com”);

    See https://bugzilla.mozilla.org/show_bug.cgi?id=382708

  14. Eric Jung wrote on

    Nevermind, I filed:
    https://bugzilla.mozilla.org/show_bug.cgi?id=513940