RJ Walsh has just completed the first revision of the add-on verification suite we proposed a few months ago, now live on addons.mozilla.org. This suite gives add-on authors and editors an automated tool to examine add-ons for common problems, and recommends fixes where appropriate.
The goals for the project are:
- Detect problems in add-ons, including issues with security, compatibility, and other areas. The author can receive alerts if problems are found.
- Help editors by automating simple, repetitive tests so they can spend their time doing more interesting things
- Maintain the level of excellence in public add-ons that Firefox users expect
Accessing the verification suite
If you are an add-on author log in to your developer control panel and choose “Versions and Files” for one of your add-ons. Find the version you want to analyze and click “No Test Results.” This takes you to the testing screen. We’ll make this process easier in a future update.
What you’ll see
After a brief analysis, a summary is displayed at the top of the page with what tests were run and their results. If you see all green check marks, congratulations! Your add-on has passed the verification suite’s current set of tests.
If you see any warnings or errors scroll down the page to see a detailed report. Beside each message that is reported is a [help] link which points to a page of documentation explaining each test in more detail.
Things to remember
Keep in mind that many tests are written to catch a broad number of cases and may display false positives. Seeing warnings in the add-on analysis does not necessarily mean your add-on has problems – it’s simply raising a flag to double check that area. As usual, we are very open about exactly what we are testing here. This tool, just like AMO, is completely open source and can be browsed online.
Over the next few months we’ll refine the tool to prevent false positives and add additional tests. If you can think of other tests that would be helpful for authors or editors please submit a patch or just let us know. We expect to have the tests running automatically when an add-on is uploaded by the end of the year.
If you have any questions, comments, or ideas for improvements please let us know, either as a comment on this blog or on IRC.