Mozilla Add-ons Community Blog
Categories: developers documentation policy releases

An Overview of the AMO Review Process

20 responses

Almost exactly a year ago, I wrote a long post detailing our add-on review process. It has been a few weeks since the launch of the new developer tools, and that warrants a new post explaining how the new review process works. I’ll try to keep it short this time.

By the way, this post doesn’t cover Personas because they have a separate review process.

Submitting Your Add-on

Did you create an add-on you want to host on AMO (addons.mozilla.org)? Just visit the AMO Developer Hub to get started. You’ll need an AMO account and you should probably take some time to read our submission policies. We also have a detailed guide that explains what we look for when reviewing your submissions and deciding if and how it should be listed on the site.

When you submit a new add-on, you will have to choose between 2 review tracks: Full Review and Preliminary Review.

For the Full Review track, our reviewers will check that your add-on is safe to use, respects users’ privacy and choices, doesn’t conflict with other add-ons or break existing Firefox features, is easy to use, and is worth publishing to a general audience. While this appears to set a very high admission threshold, it really doesn’t; our goal is to publish as many add-ons as possible, while offering safe and useful choices to our users. Add-ons that receive Full Review approval will appear as Public, have the option to be Featured, can enable Contributions, and in general get much more public attention and downloads.

The Preliminary Review track only requires your add-on to be safe to use. Reviewers look for issues that could put the users’ system or data at risk. Add-ons with Preliminary Review approval appear on the site as Experimental, can’t be featured, and get lower search rankings. Like Public add-ons, they have automatic updates enabled.

So, why would anybody choose Preliminary Review? Many authors have expressed to us the need to host their add-ons on AMO in an intermediate state where they don’t get as much visibility and don’t have their add-ons as heavily scrutinized as they are for Full Review. Also, it serves as a replacement for the sandbox that we had in the previous review system. Now, if an add-on doesn’t pass Full Review, it can still qualify for Preliminary Review if it passes the basic security checks that we do. That will at least allow the add-on to be visible, installable and updatable.

Add-ons that don’t pass Preliminary Review are considered unsafe and are not published on AMO at all.

When your add-on is reviewed, you should receive and email message form us explaining if the review was approved or not, and what kind of approval you got. The message should include notes from the reviewer explaining what was discovered during the review and any issues that need to be addressed. Make sure that your registered email address on AMO is active and that you check on it frequently. On occasion we’ve been told that review email end up in the Spam or Junk folders, so you may want to check there if you think you should have heard from us by then.

Changing tracks

If your add-on has been approved in the Preliminary Review track, you can nominate it to the Full Review track after a 10 day waiting period. The 10 days begin from the time of your last Preliminary approval. After that you should see a Full Review nomination option in the Status & Versions page in the Developer Hub.

Updating your add-on

You can upload updates for your add-on from the Status & Versions page in Developer Hub. All updates are reviewed as well, and they will receive Full Review or Preliminary Review depending on the review track your add-on is in. If your first submission wasn’t reviewed before you uploaded the update, the new version will take its place in the review process.

Updates for add-ons in the Full Review track that don’t pass Full Review can still get Preliminary Review approval if any issues are found in the submission. In this case, the update will not be pushed to users, but it will still be visible and installable in the All Versions page of the add-on. The add-on remains in the Full Review track, and a fixed version needs to be submitted before updates can be pushed to users.

Update reviews are normally very fast (under 5 days), but we understand sometimes an update needs to be pushed as quickly as possible. Security fixes and broken external dependencies are the most common reasons for requesting this. In those cases you should contact the AMO Editors directly via email or IRC, and state your case. We can normally respond within a few hours.

The Review Queues

We have 3 lines where all review submissions are queued: one for Full Review nominations, one for Full Review updates, and one for Preliminary Reviews (both nominations and updates).

The waiting times for these queues can often vary, but our goal is to process all updates and preliminary reviews within a couple of days, and Full Review nominations within 10 days. I post queue state updates every week in the Add-ons Forum if you’re interested.

At this moment we’re doing very well on all queues except for Full Review nominations. We have a very large amount of reviews to process caused by the transition from the old review system to the new one, since thousands of old add-ons that never passed our reviewed process now require to do so in order to live on AMO. We expect to bring that queue back to normal within a few weeks, though.

AMO Editors

The Mozilla add-on review team is called AMO Editors. It’s composed mostly of volunteers; experienced add-on developers who take time from their busy lives to ensure the add-ons we list are good to use. Reviewing add-ons is a meticulous and often monotonous task, and many of them spend several hours a week working on this for free.

Reviewing add-ons for AMO is no small feat. Just last month the editor team performed over 2400 reviews. While these are exceptional times because of the release of Firefox 4 and the launch of the new review process, the average monthly review count usually hovers around 1000 reviews. With hundreds of submissions pouring in every week, there’s always work to do. So, if you run into an AMO Editor, give him a hug, or buy him a beer :).

Questions and comments welcome, as always.

20 comments on “An Overview of the AMO Review Process”

  1. Tom wrote on

    Hi there – I’d be interested in hearing a bit about what exactly an AMO Editor does during a Full Review for new Addons and updates.
    I imagine that some sort of validation script is run to identify common issues in the code. Do editors go through all of the code manually as well? Do they run the addon and do some preliminary testing? Or is it all code-based reviews? I guess I’d like to know what goes into a typical review; that might affect how giant of a hug or beer I should give an AMO editor 🙂

    1. Nils Maier wrote on

      During preliminary review an editor will at least:
      – Run the code validator and inspect the warnings
      – Do a separate code review of all code. This is not meant to catch minor bugs, but focuses on issues concerning security, data loss and other grave stability problems.
      – Sometimes the add-on is tested in order to ensure that complicated code was assessed correctly.

      During a full review an editor will at least:
      – Do the preliminary review
      – Code review additionally for compatibility with the application (Firefox for example) and other add-ons in general, performance and other minor errors.
      – Check that all policies are followed, especially concerning user’s privacy, No Surprises (no hidden changes to user settings and such) and code quality.
      – Install the addon and test the primary features, not only if they work as expected, but also if there are runtime error that need to be addressed. The add-on is usually tested in different applications depending on the compatibility flags it gives, for example Firefox 3.6 and Firefox 4 and Seamonkey.

      Furthermore the job of an editor entails giving detailed help and additional feedback to add-on developers as needed, explaining safe alternatives to unsafe code, explaining our policies and why a particular policy exists in the first place, looking out for new classes of issues and discussing new and updated policies, and either receiving anything between a warm “thank you” to furious rage and personal insults from add-on authors.

      More information is available in our Review Guide, which is meant to guide new team members and as a reference for all other editors:
      https://wiki.mozilla.org/AMO:Editors/EditorGuide/AddonReviews
      This document is currently somewhat in flux as the last inconsistencies and uncertainties that arose because of the new review process are ironed out.

      1. Tom wrote on

        Thanks for the detailed info. That is really helpful to see all that the editors really do to ensure that an addon is of acceptable quality.

        Will definitely have to spring for the 24-ounce beer next time I run into an AMO editor 🙂

  2. SpeedGun wrote on

    I have a small question that can perhaps be answered here:

    What if you have a addon that has been “preliminary reviewed” and you want to push it to a “full review” state, how do you do this?

    1. Jorge wrote on

      The “Changing Tracks” section above should have the answer for that.

      1. SpeedGun wrote on

        Ah, missed that part. Thank you.

  3. Kissaki wrote on

    I love it.
    Also the informative note on why an addon is not approved, what you have to improve.

    Thanks for all your hard (and continued) work reviewers and all!

  4. Kohei Yoshino wrote on

    A Japanese translation of this article is available at
    https://dev.mozilla.jp/2011/02/overview-amo-review-process/

    1. Jorge wrote on

      Thank you for all your hard work!

  5. Ken Saunders wrote on

    “I love it.
    Also the informative note on why an addon is not approved, what you have to improve.

    Thanks for all your hard (and continued) work reviewers and all!”

    I was going to say the same (especially the thanks part).

    It’s extremely helpful and appreciated when there are suggestions for fixes.

  6. Thiago Ganzarolli wrote on

    My addon called Encontre Seu Pacote is waiting on the Full Review queue for quite a while, and in the meantime it has disappeared from AMO. You should have left a warning page, telling the customers that it’s disabled for the moment. It was a very bad choice to do it so harshly.
    On top of that, nobody gave me a deadline for the process, and in the page where it is listed (just for me, when I am logged in), there is a message threatening to delete my add-on, as follows:

    “This add-on will be deleted automatically after a few days if the submission process is not completed. ”

    However I completed the submit process! I am extremely disappointed with Mozilla.

    Some people here are complimenting you for the huge effort, and I admire that, but why change the process so suddenly and in a form that slows innovation? Apple store syndrome?
    And please, more feedback for developers.

    1. Thiago Ganzarolli wrote on

      Update: although the review process is not yet completed, my addon page came back and is appearing again in AMO, also the threatening message is gone 🙂

  7. Jivko wrote on

    “Update reviews are normally very fast (under 5 days), but we understand sometimes an update needs to be pushed as quickly as possible.”

    I hate the period when Firefox updates.Reviewers are not organized or not enough.The update of my Bloody Red theme is in sandbox for a week or more and there’s no word from a reviewer.I updated my themes since January 2011 for Firefox 4 but RC 1 came with small changes and some bugs appeared even though it was written that there won’t be any changes after beta 7 on this blog.I had to upload new versions.

    1. Jorge wrote on

      “not enough” is exactly right. The period around Firefox updates inevitably slows down reviews. However, we have been keeping up with updates, and most are reviewed within a week. I see yours near the top of the update queue and it should be reviewed very soon.

  8. Jivko wrote on

    Thanks for the quick reply.Looking forward the review. 🙂 🙂 🙂

  9. New AMO User wrote on

    (I may just have to test it, but) Is there any way to upload a test/fake Addon & select “No Review”? (as in it won’t get placed in the Review queue & also won’t be downloadable by the Public, but should show up to myself, when logged in). I don’t want my testing of AMO’s interface to hinder the Review process. I’m new to AMO & I wanna test the interface, before I upload any real Addons. Also, has anyone made a Tutorial/Demo (YouTube video) of the AMO Addon Management interface/process? I tried to search for some, but couldn’t find any.

    1. Justin Scott (fligtar) wrote on

      You can submit to our staging server at http://addons.allizom.org and it won’t be reviewed.

      1. New AMO User wrote on

        I saw some links to allizom.org URLs before & I was wondering, does that share the same login?…or do I register there too? Also (& not that this is a bad thing), but wouldn’t the testing I do over there get clobbered/reset periodically?…like whenever they are testing a new Addons interface/site…doesn’t it get scrapped/reset from time to time?

        Since I have your attention, can you (or someone) look at this topic in the Forums?

        This Forum loses caps in Usernames / phpBB Search is Useless
        https://forums.mozilla.org/addons/viewtopic.php?f=26&t=3148

        (Don’t know if I can make the text the link {instead of the URL} here.)

  10. Jim wrote on

    Can you publicly publish all of the queues? This would be valuable for admins, users, and developers. Thank you.

    1. Jorge Villalobos wrote on

      This has been discussed before, and we believe that this isn’t a good idea. Critical security updates could be in the pipeline, unreviewed add-ons would get more exposure than they should, and all sorts of noise would arise from the unordered way in which the volunteer team handles the queue.

      We don’t see any clear benefit in exposing the whole queue to the public. Developers can now see the position their add-on holds in the queue, and they are informed of any status changes in their add-ons. Feel free to post in the bug if you disagree.