The Add-on Review Process and You

31

The add-on review process remains a mystery for many add-on developers. As a developer myself, I admit it feels like dropping your add-on into a bottom-less pit and just waiting (and hoping) for something to happen. As the weeks pass by, patience runs out and you wonder what’s going on.  Developers have rightly demanded more transparency in our review process, and the purpose of this post is to explain where we are now, and what we’ll doing in the future to improve it for all add-on authors.

Starting next week, I’ll be posting regular updates on the state of the queues and the review process in general. This post explains how the overall process is functioning now, which will allow me to keep the updates as concise as possible. Whether you’re a new add-on author, a veteran developer, or an add-on enthusiast,  I recommend you take the time to read this and ask any questions you think are unanswered here.

The Review Process

AMO uses a sandbox system. All add-on files uploaded to AMO are placed in the sandbox, where only a minority of users can locate and install them. They are currently labeled as “Experimental” on AMO listings, but may be re-labeled to “Unreviewed” or something similar in the future.

All uploaded files are scanned for malware (viruses, trojans, etc.) using a variety of tools. A recent security problem exposed a vulnerability in our system, and that was that we were relying on a single virus scanner which wasn’t all that good. Our scanning system is now more robust, and we’re considering other ideas on how to improve security in the sandbox system.

In order for an add-on version to become public and readily available to all, it needs to be submitted for review by its author, and pass the review process.

For new add-ons, which we usually call nominations, the review process goes as follows:

  1. The author goes to the Developer Hub and creates a new add-on listing by uploading the add-on file and adding information such as descriptions and preview images.
  2. The uploaded file is run through our Code Validation Tool. This tool warns the author about possible code quality issues which may be reason enough for rejection. The Validation Help page explains in detail the reasons behind the checks and which are more important to pay attention to.
  3. The author can nominate the add-on as part of the submission process, or later from the add-on status page accessible through the Developer Hub. Having descriptions and at least one preview image are requirements for nomination. We no longer require a minimum amount of user reviews for nominations, but having some may improve chances of approval.
  4. After nomination, the add-on status page will indicate a status of “In Sandbox; Public Nomination”. This means the add-on is in the nomination review queue.
  5. When the add-on is reviewed, it will either be approved, denied, or additional information may be requested from the author. An email sent to the author will contain in detail what the reviewer though about the quality of the add-on and possible areas for improvement. In case of rejections, these notes should all be taken care of before nominating the add-on again. The approval message may also contain suggestions, and even requirements for the next release, so authors should always read the message entirely.

Approved add-ons will have a new “Public” status; denied add-ons will have an “In Sandbox” status. Authors can reply to the email in case they want to discuss the notes directly with the reviewers. Thank you notes are also appreciated :).

After an add-on becomes public, new versions of the add-on (updates) undergo a similar process:

  1. The author goes to the Developer Hub and creates a new version of the add-on by uploading the add-on file and adding version information.
  2. The uploaded file is run through our Code Validation Tool. This tool warns the author about possible code quality issues which may be reason enough for rejection. The Validation Help page explains in detail the reasons behind the checks and which are more important to pay attention to.
  3. The new version will be automatically nominated for the public (except for some bugs). The status of the latest version can be easily verified in the My Addons page, in the Versions and Files section. It should be “In Sandbox; Public Nomination” at this point. If it isn’t, let us know.
  4. When the update is reviewed, it will either be approved, denied, or additional information may be requested from the author. The email sent to the author will contain in detail what the reviewer though about the quality of the add-on and possible areas for improvement. In case of rejections, these notes should all be taken care of before updating the add-on again. An email sent to the author will contain in detail what the reviewer though about the quality of the update and possible areas for improvement. In case of rejections, these notes should all be taken care of before submitting a new version. The approval message may also contain suggestions, and even requirements for the next release, so authors should always read the message entirely.

Approved versions will have a new “Public” status and will be automatically pushed to users that have the add-on installed. Denied versions will have an “In Sandbox” status. Authors can reply to the email in case they want to discuss the notes directly with the reviewers.

At any point in this process the author can send a message to amo-editors AT mozilla DOT org to request information about their add-on review status.

The Review Queues

A common concern for add-on authors is “How much time will I have to wait for my nomination / update to be reviewed?”. This is difficult question to respond, given how different add-ons are and how the reviewing team works.

As explained before, all add-ons nominations and updates are placed in a review queue. There’s a queue for nominations and a separate queue for updates. They are both sorted by waiting time, the oldest pending reviews being at the top. They are reviewed by a team of volunteers (mostly), who give higher priority to oldest reviews. Reviews have wildly varying levels of difficulty, though, and that’s the reason that a fraction of them take specially long to be done. The most complex ones are reviewed by a senior or admin editor, usually me. If I reviewed your add-on in the past couple of months, chances are that it is much more difficult to review than the norm.

I post queue status reports every Friday in the Add-ons Forum. In these reports you can see a breakdown of each queue by waiting time ranges, and showing the progress we’ve done in the previous 2 months. Our current goals should be clear from the reports: all updates should be reviewed within a week, and all nominations should be reviewed within 2 weeks. The latest report shows we have pretty much achieved this goal for updates, and we still have work to do for nominations. Compare these numbers with the very first report I posted last September, add to this the fact that Firefox 3.6 was released recently (lots of new add-ons and updates), and you’ll realize just how much things have changed with the queues in the past 5 months.

As a little bit of trivia, in September there were approximately the same amount of nominations waiting for longer than 4 weeks (305!) than the total amount of add-ons we had last Friday in both queues combined (311). Today, updates are normally reviewed within a couple of days, and nominations are normally reviewed within 3 weeks. And all of this is improving every day, thanks to…

The AMO Editors

Add-on reviews are performed by the AMO Editors team. Everything you ought to know about us should be accessible from that link. If there’s more you would like to know, please contact us and we’ll update the wiki accordingly.

From the wiki:

AMO Editors is a Mozilla community dedicated to guard the security and reliability of add-ons listed on AMO. As part of the add-on review process, editors review the code and functionality of new add-ons and add-on updates, and decide if they’re appropriate for publication or not. Editors follow and enforce the established AMO Policies.

Most editors are volunteers; experienced extension developers that want to participate in the review process and improve add-on quality and security on AMO. Some of us are Mozilla employees, and a few others have been contracted to help us in our effort to bring sanity back to our queue lengths. We have 74 registered editors, but only a little over a dozen are active at any given time. We receive many applications for new editors, and a new member is added to the team almost every week.

There’s some information in the AMO Editors wiki page that authors will find very valuable:

  • Useful information for Editors. This page explains in more detail how we perform our reviews, and some additional reviewing guidelines that may not be specified in the official policy documents. This page and the pages it links to are the basis of our editor training process.
  • Useful information for Add-on Authors. How to tell if your add-on is in the queue, tips on how to make sure your add-on is reviewed quickly and many useful reference links.
  • How to Become an Editor. Our doors are open and we appreciate your help :).

Most of this information was added to the wiki fairly recently. I’ve been trying to figure out how to better expose it on AMO, and I think this blog post is a good start. There are a few places where I think it will be useful to link to this blog post.

The Future

For the sake of transparency, authors are requesting us to open up the review queue and make it fully visible to the public. As you’ll see in my comments on that bug, I don’t think that’s such a good idea. As I explained before, add-ons are not reviewed strictly in order. Some may remain in the queue for longer times while others are reviewed instantly, depending on several factors. We already receive plenty of complaints about this in the mailing list, and I think that opening up the queue will multiply these complaints significantly, specially considering that some add-on authors are very competitive (sometimes businesses dedicated to their add-ons). Since we have to manage the mailing list as well, I think our work will be frequently interrupted by “why X and not me” messages if we chose to do this.

However, there are a few things that I think we can do right now that will help improve the queue visibility situation. I’ve opened them up as separate bugs:

Ideas are welcome of course. Please comment on the bugs that are important to you. Open new ones if you think there’s something else that needs fixing. Comment on this bug, or send me a message to jorge AT mozilla DOT com. Above all, let’s keep the communication as open as possible.

Oh, and congratulations if you made it all the way to the end of this post :P

Tags: , , , , , ,

Categories: developers, documentation, end users, general, policy

31 responses

  1. Matt Kruse

    I would like to suggest a way to flag an update as “urgent”.

    For example, if a security bug is found or there is a problem that causes the add-on to malfunction for some users. In this case, a quick fix might be needed and a wait of a week or two just puts users at risk or annoys them while their add-on is broken.

    One of my add-ons is an extension to customize the Facebook interface. After the recent Facebook layout change, the add-on become completely useless. As new users downloaded it, they wrote negative reviews saying it was broken and didn’t work at all.

    I had already uploaded an update that resolved the problem, but they have no way of knowing that. Meanwhile my download counts go down, my add-on rating goes down, users get frustrated and removed the add-on, and bad word of mouth spreads. Even though the problems were already fixed and just waiting for a minor update to go through the queue!

    I appreciate all the efforts by everyone involved in this process, and on the whole it works great. I would just like some way to expedite the process in some situations. Hey, I’d even be willing to pay money to go to the top of the queue. Consider that ;)

    Thanks!

  2. Jorge Author

    @Matt Kruse: for cases like this I recommend you send us a message to the mailing list explaining your case. More often than not we’ll review your add-on on the same day.

  3. Andraz Tori

    I’d second Matt’s idea about “urgent” flag with a simple field to explain why it is urgent.

    Just recently Zemanta’s extension exposed a bug in Ubuntu’s version of Firefox, making it unusable. We quickly fixed it and new version was quickly approved, so no harm was done.

    However I think the approval process should by-default support such an urgent event, since it probably happens often enough.

    Andraz Tori, CTO at Zemanta

  4. Brett Zamir

    A couple things:

    1) Slightly off-topic, but what happened to the ability at AMO to challenge user reviews? Was it just because there were too many to review?

    2) I was always wary of thanking you folks because I’m afraid of it being perceived as spam (i.e., if many people receive the AMO-editors email and only one person did the review).

  5. Jorge Author

    @Brett:

    1) It’s still possible to reply and flag user reviews for moderation. It’s just little hidden in the full user reviews page. See for example:
    https://addons.mozilla.org/en-US/firefox/reviews/display/5235

    2) It would be spammy if we received thank yous from everybody, but it’s nice to get one every now and then :)

  6. Brett Zamir

    1) Cool, thanks!

    2) Well, how about a thank you here then? :) It is awesome how you manage to get through all of those add-ons, and still giving constructive feedback as I know I’ve received.

    One new throw-away suggestion… It’d be cool to see AMO offer quick links in the developers pages to do things like add a new version without having to navigate through a number of (slower https) pages to get there.

    Thanks again…

  7. Marko Samastur

    How about a paid track accompanying free one that would pay for professional reviewers?

    Our company would gladly pay for a better service, where we could get updates through more promptly and pick a date on which they appear.

    I am certain we are not the only ones who would want that.

  8. Jean

    Single home user here, on board with addons since 0.6.5
    “herding cats” comes to mind when I read these review process articles :-)
    Congratulations and deepest appreciation for the entire team’s ability to continue balancing user’s experience and security against the pressure for publication from developers.
    I always deal with individual developers wrt donation and requests, so this is no blanket criticism of developers :-)

  9. Bee

    Hi!!!!!!!!!!
    What about bugs blocking updates?!!!!!!!!!! I know BeeFree is blocked because of a bug in AMO!!!!!!!!!!!!!

    @Brett: you could add to your firefox bookmarks a new “Addon” folder within a link pointing to: https://addons.mozilla.org/en-US/developers/versions/add/ADDON-ID
    No way is better than that!!!!!!!!!!!!!!!!!!!!!

    bye!!!!!!!!!!!!!!!
    ~bee!!!!!!!!!

  10. Ken Saunders

    Ok so I made it all the way to the end of the post (for the most part) without cheating. What do I win?! :)

    Fantastic post by the way.
    Thanking editors is common courtesy so it’s no brainer. Everyone likes to know that their time and work is appreciated and I suspect/doubt that most people have any idea or take the time to consider what goes into reviewing an add-on.

    The review process is (no doubt) more comprehensive than it was when I was a reviewer for a short time (pre Remora) and back then it was a lot of work. Not hard work, but a lot. Perhaps you should enlighten people on the actual process.

    Tell them that not all editors have the ability to test a particular add-on on all 3 major OS’s, that some add-ons require that the editor has to create website user accounts, accumulate, add, move, and collect data such as bookmarks, post updates to social media, compose and publish blog posts, customize interfaces with the add-on’s full capabilities (for the most part), check out all of the add-on’s user options and access points, create new Moz product profiles, restarts, and on and on. And that’s just basic functionality. And if those things aren’t being done by editors, then they should just quit and go work for Toyota. :|

    This post should be referred to somewhere in the How-to-Library which by the way could use a getting your add-on on AMO section. I could be wrong, but the How-to Library doesn’t include info on what to do with an add-on once you build it.

    I really like the idea of renaming the experimental label.

    One other AMO related thing.
    It would be nice to see the developers/addon/edit page redone.
    Move the content that’s in the right side bar and make it all a single page of options/access points. It would reduce some clicking/page loading and waiting so for example, Versions and Files would have Upload New Version right under it.

  11. Dr.S.

    What’s all this discussion about?

    When it takes several pages to explain the review process than I guess that there is something wrong at best.

    Just implement a transparent handling of the nomination/update queue on a first come first serve basis.

    This way there can only be one complaint: It takes to long.

  12. Jorge Author

    @Marko Samastur: we’ve heard that suggestion before, yes. Our goal is to reduce waiting time enough that it won’t be necessary to have this. If you need special timing for a release, you can contact us about it (not the day before, of course).

    @Bee: your add-on is in the queue, there isn’t any bug affecting your add-on.

  13. Bee

    Hi Jorge!!!!!!!!!!! though I’m unsure if you’re the same Jorge who isn’t answering to my mails!!!!!!!!!!!!!!!!!!!!!!

    Well, however, i know BeeFree isn’t in that queue and there is an evil bug blocking it!!!!!!! All addons have got their updates, but beefree!!!!! this is quite the proof i need to suspect of flaws in AMO!!!!!!!!!!!! This is neither the first time of bugs blocking updates!!!!!!!!!!!!! There’s no other way to explain what is going on!!!!!!!!!!!!!!!
    BeeFREE is blocked, yet i’ve found it marked as “experimental” here: https://addons.mozilla.org/en-US/firefox/addons/versions/57131 so, i did its upload on Feb 5, a dozen of days ago!!!!!!!!!!!!!!! It’s marked as “public” but it isn’t public, indeed it’s “experimental”!!!!!!!!
    BeeFree isn’t in any queue, or else it would have been published!!!!!!!!!!!!!!
    Too many addons have been published in the same stretch of time!!!! some of them have got two updates!!!!! beefree’s not even one!!!!!!! a lot of time passed by, and if the turn of BeeFree never comes, it’s because there is something of broken in AMO!!!!!!!!!! it’s quite obvious!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    Then, I filed a bug report….. just to realise that Mozilla is ignoring it and it’s keep ignoring bugs!!!!! but this won’t help!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    I did three uploads of three different versions of beefree!!!!! Every time, every upload, went wrong!!!!!!! Because there is always a new bug at AMO!!!!!!!!!!!!!!!!!!!!!(YEAH!! ..looks like, all bugs ain’t honey bees!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!)

    bye!!!!!!!!!!!!!!!!
    ~bee!!!!!!!!!

  14. Bee

    «Our company would gladly pay for a better service, where we could get updates through more promptly and pick a date on which they appear.»

    That’s a very bad idea!!!!!!!!!!!!!!
    What nobody needs is to create discrimination!!!!!!!!! NO ADDON LEFT BEHIND!!!!!!!!!!!!
    A toll highway for who can pay, and all the others behind!!!!!!!!
    Those capitalistic ideas are what is ruining the world!!!!!!!!!!!!!!!!!!!!!!
    It’s like when Mozilla receives “donations” from Google!!!!!!!!! It’s the reason why we’ve got a biased Firefox browser!!!! it’s why the default search engine of Firefox is Google!!!!!! Regarding IT, Google is the worst and most evil multinational company ever!!!!!!!!!!!! but then, again, why is Google the default search engine for firefox?!!!!!!!!
    ANSWER: If Mozilla was a a political party, then i’d to call Google’s donations: BRIBES!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    The same thing would happen if AMO turns commercial!!!!!!!!!!!!!
    AdBlock Plus and BeeFREE would be the very first victims!!!!!!! On the other hand addons showing adverts, made to take away the privacy of users and with the help of economic supports (from evil, local, small, big and multinational companies) would be the very first to have their daily updates published!!!!!!!!!!!!!
    Those updates would turn into new daily earnings for AMO/Mozilla, but then you need to ask yourself: wouldn’t be that another loss for everyone’s freedom?!!!!!!!!!!!!!!!!!!!

    bye!!!!!!!!!!!!
    ~bee!!!!!!!
    http://honeybeenet.altervista.org/

  15. Ken Saunders

    Bee, I understand that you’re frustrated and obviously angry (judging by the exorbitant amount of exclamation marks), but posting a tirade about Mozilla is certainly unnecessary and very insulting to me and everyone else that freely supports, and is passionate about Mozilla.

    Someone trying to bash Mozilla over Google being the default search provider is always a trigger point for me personally, but I can leave that alone because this is not the proper venue for such a discussion, so please stick to subject matter that pertains to the topic of this post, or perhaps better yet, start a new (non anti-Mozilla) thread in the AMO Forums.

    For the record, it’s Firefox, not FireFox.

    “BeeFREE is a FireFox’s addon”

  16. Bee

    Hi Ken!!!!!!!!!
    You’re misunderstanding my exclamation marks!!!!!!!!!!! I ain’t angry and i didn’t meant to insult anybody!!!!!!!!!!!!
    I don’t know about that frustrated thing, but a little bit i could be!!!!(lol)
    That’s because at AMO they just don’t care about Bee Free!!!!! So, they forgot to look at it, and to fix all those bugs blocking it!!!!!!!!!
    but i ain’t angry for that!!!!!!!!!!!!!!!! i’m very calm!!!!!!!!!!!!!!!!!!!!! though i don’t very approve what they’re doing!!!!!!!!!!!!!
    I’ve little and no reasons to bash people working for free at AMO and Mozilla!!!!!!!! They’re doing a good, great job!!!!!! but those paid (with google’s money, for instance) weren’t supposed to work for all?!!!!!!!!!!!!!!!!!!! why don’t they care of beefree?!!!!!!!!!!!!!!!

    bye!!!!!!!!!!!!!!
    ~bee!!!!!!!!!

  17. Pikadude No. 1

    “i’m very calm!!!!!!!!!!!!!!!!!!!!!”

    I *love* the irony of that.

  18. Pavel Cvrček

    Jorge, thanks for great post. As AMO editor which was active for a long time I think that such posts are very useful. I don’t know if it’s still problem but one or two years ago there were many add-ons in review queues which was very hard to review. Typical problems: unknown language, add-on requires registration on some server, binary components etc. Maybe you should write post “How to help editors” or something like this.

    Btw. such a posts are great but I’m not sure if new developers find this post. Maybe link to this post from Developer Hub/upload form?

  19. Jorge

    @Pavel: thanks! We have mitigated the problem with difficult reviews to a very manageable level, and I’m working with the localizer community when it is too difficult to assess an add-on that is too region-specific. Regarding giving this post more visibility, I’m already working on this: https://bugzilla.mozilla.org/show_bug.cgi?id=541221

  20. Josh WiseStamp

    @Jorge Thanks for explaining this and for the great and amazing work of all the editors! things have defiantly changed in AMO to the positive side since you came! way to go and keep up the great work
    Josh @WiseStamp

  21. Jivko

    Very useful information for developers.I hope that the add-on review thing will take less time in the future.For example my Sky Pilot Classic theme is in sandbox for two weeks and I hope that it will get out soon.

  22. Scott

    I suggest that an article like this one include a byline. Reading through it, there is a definite voice writing the article (it, for example uses the word “myself”), but I could not at first glance tell who that person was in the ensuing set of comments. I went back to the link that took me here, and so could tell that Jorge is the author. However a name after the title would have made it simpler.

  23. john

    I don’t really mind getting one more account, but I think it would be good if you also stated that I need to register for experimental add-ons.

    (I know the previous paragraph on the quoted view says something about “public add-ons” but anyway…)

  24. Jorge Author

    @john You don’t need to log in in order to install experimental add-ons.

  25. Steffen

    Hi Jorge,

    maybe I missed something… Anyway:

    As far as I understood the Add-On philosophy new extensions need to be reviewed for new let’s call them Firefox version classes. When I have an Add-On for Firefox 3.6.* I need to package another add-on package for 3.7.* and that needs to be reapproved. I guess 99% of such ‘version step packages’ will still work. How does this reflect in approval process? Do you compare the code of two add-on versions and honour if only the install.rdf changes? Or will there be flag in future or another enhancement coping with that?

    In that context: It would be very cool if the Firefox versioning schedule would be transparent and if I could subscribe somewhere so that I do not miss to version number upgrade my Add-On before earning a message “this is an older add-on” on my add-on listing.

    Thanks,
    Steffen

  26. Jorge Author

    @Steffen: if a new version of Firefox comes along and your add-on works on it without any code changes, you can go to the AMO developer tools and change the add-on compatibility information without having to upload a new version or go through the review process. Your add-on will be automatically updated and will be offered to users of the new Firefox version.
    Regarding notifications, there are several Mozilla blogs you could follow to know when a new version comes around. This one is one of them. We will always notify developers in advance about new major releases of Firefox, with the necessary information to keep add-ons up to date.

  27. If

    What preview images should extensions use when they have no UI? For example those that just want to add a keyboard shortcut?

  28. Jorge Author

    Anything will do in those cases. It could even be some sort of logo for the add-on.

  29. UlyssesBlue

    I would like to see a minor change to the rating system for add-ons. The amount of stars something has can be misleading.

    Let’s consider the example of a really good 5 star rated add-on which releases a new update which has some really major faults. All subsequent reviews are 1 star, but the rating says 3 stars, implying it’s still quite good. Then there’s the opposite case of a poor add-on which suddenly releases a new version which is a major improvement.

    In both these cases it would be worthwhile being able to view both the overall rating, and the rating for the current version, so as to better understand the quality of what you’re getting.

  30. wanderingone

    Hi,

    I recently installed the BeeFree add-on for Seamonkey and when I pause the mouse on links it gives certain information regarding
    the target, tracker and original, but when the cursor is paused over the BeeFree icon in the systray it says ‘BeeFree is in an unknown status’.
    I’m just wondering what this might mean?

    I tried enabling the Debug option, but it doesn’t seem to make much of a difference that I can see.

    Any advice would be appreciated.

    Thanks

  31. Nemu

    um i dunno everything else, but one thing i’ve noticed (relating to reviewing Mozilla addons) is there is no way to quickly check back on your previous comments/’reviews’.

    sure, you can view them all from your profilepage (as can others) but even when you click the review itself, all it takes you to is the addon page, and if your review is not the very last review, or at least somewhere on the first page (when click all reviews for addon), you’d go through page by painstaking page to look back until you find one that’s yours,–there is a link to see all previous link (as well as informing you you have reviewed before) but not until then.

    ironically there is a little hash sign to get the review all by its little lonesome, but no ‘button’ to view all previous reviews (plus developer replies..)..that’s..=\