Mozilla Add-ons Community Blog
Categories: developers

Burning down the Add-on Review Queues

13 responses

A long standing problem with AMO have been the add-on review queues and waiting times.

For those unaware, we have a review system that all new add-ons and add-on updates have to go through before they are available to the general public. The system is a combination of automatic code analysis and manual code and feature testing. The manual side is handled by a dedicated team of volunteers, known as AMO Editors. We’re a small group that has had to handle a massive amount of work, specially since the release of Firefox 3.5. We had spent the last year catching up with the numerous new submissions and updates, and the numbers will only grow once 3.6 is out.

Just 2 months ago, the new add-on review queue had over 600 pending add-ons, and the update queue had more than 250. Both had several months of waiting times, and add-on authors were increasingly unsatisfied, with good reason. Many trivial or important updates have had to wait a long time before seeing the public, causing users to complain and give add-ons bad reviews for the slow reaction of their authors.  That’s just unfair to add-on authors, and we needed a solution.

I joined Mozilla recently with a very clear objective: to reduce the review queues back to a manageable size. I have helped implement several solutions that would improve the state of affairs, like posting detailed weekly queue reports in the AMO forums, and temporarily hiring professional add-on developers to help with the burden. We estimate that our current volunteer team is large enough to keep up with the inflow, so now we just need to reduce queue size down to our ideal waiting expectations (2 weeks for nominations and 1 week for updates). After that I hope we will regain stability.

I’m happy to announce that our efforts have paid off. The latest queue report indicates that we’re down to 301 nominations, and 120 updates. That’s half of what we had 2 months ago! Just last month we reviewed over a thousand files, and our pace is increasing every week. About half of our pending reviews are within our ideal waiting time frames, and many of the ones that have been waiting for longer require an admin review for various reasons. Things are looking great.

There’s still lots to do in order to improve add-on author experience, and that is an ongoing effort. I’ve received quite a few suggestions on how to improve the reports, and I hope to implement those soon. If you’re a developer and you have anything else you’d like help with, don’t hesitate to contact me 🙂

13 comments on “Burning down the Add-on Review Queues”

  1. Brian Fernandes wrote on

    I know there were ~ 470 add-on nominations when I nominated mine and I thought it would take 3 months or more; mine was reviewed in under a month. Outstanding work! This is going to make a significant difference to the overall add-on experience.

  2. Alex wrote on

    Comment has nothing to do about your latest entry but I thought I heard that Thunderbird was suppose to e updated this past week. What happened or war this rumour just bull?

  3. Alfred Kayser wrote on

    It is indeed improving, but still a large backlog to work away, and I really appreciate the hard work here.
    My Nautipolis for Seamonkey is now 5 weeks in review (posted October 6th). Considering that this is a theme (which requires less strict checking), and that it is a old friend of Mozilla (since 10 years now) and Firefox (since 5 years) it should not to be too hard to get it reviewed…

  4. Mark Smith wrote on

    Thanks for the informative post. There is a lot of frustration on the part of add-on developers and publishers. Is there a way to tell if an extension update is in the review queue?

    For example, I submitted an update for Pearl Crescent Page Saver Basic 6 weeks ago and it has not been approved.

  5. Jorge wrote on

    @Mark Smith: in the developer tools, you should see a Files and Versions section, where you can see the current status for all versions of your add-on. If your last version has a status of “Sandbox, Nominated for the Public”, then it means it’s on the queue. If it only says “Sandbox”, then you didn’t nominate it when you uploaded it, or it was rejected (but you should have received an email in that case).

  6. Brian King wrote on

    Great work Jorge!

  7. Alfred Kayser wrote on

    Hi Brian, thanks for approving and nominating my themes!
    AMO is just great!

  8. Axel Grude wrote on

    The reviewing queue is a bit like the door to the law in Kafka’s short story “Before the Law” – you never know when you’re extension will be let 😉

    Anyway if you are really interested in cutting down the review queue I would ask to also look at Fx’s little brother Thunderbird, it seems the review queue there suffers because of the new Fx Release.

    My personal viewpoint on releasing is “release early and often” which leaves my extension with 4 (!) updates that have not yet been reviewed, the latest one offering an ever increasing feature set and lots of desirable bug fixes – and just today I have released the 5th one [which now includes support for 2 more hosts and 3 more locales, compared to the last published version]; I do not know whether this actually has again reset the waiting time for my queue – but in general it seems to boil down to a publishing cycle of roughly 1 published update / month (usually I integrate several bugfixes during that time, so it is not exclusively negative).

    – all thanks to the diligence of the users who constantly come up with ideas, requests and bug reports. There is an highly interesting article “the cathedral and the bazaar” by Eric Stephen Raymond (see catb.org) which points out the difference between commercial and open source software release cycles, which comes to the conclusion that a higher release frequency is indeed a good warranty for high software quality standards – obviously it also generates a lot more administrative work; I think one of the methods to cut down on the amount of work would be fairly obvious – people who have reviewed an extension before (and wish to do so) should be preferred as reviewers for updates of this.

    In my case I have a fairly complex extension with a lot of added value for long term users, but this is not obvious at first glance to a casual user – the more you use this extension the more useful features you will discover; for reviewers, testing this functionality can of course be a daunting task – you need to set up test folders, do drag / copy operations on emails, customize the extension’s interface and so on and so on…

    How much easier is it if you already use the extension and you might even be interested in what’s new – wouldn’t it be great if the reviewers would be notified of updates of any extension they have installed on their system and have reviewed previously – surely somebody who uses a piece of software daily is somewhat of an expert.

    I would propose a “mentoring system” where AMO reviewers could voluntarily take some of their favorite extensions and be contactable by the authors about important updates?

    In emergencies (such as showstopper bug fixes) I have done so before, using more oblique methods such as preoject owners mailing lists and ICQ, but it would be great if there was an official method openly available to developers of extensions that have already been published.

    The other thing that would surely diminish the problem of “stale” updates would be a more obvious presentation of newer versions on the AMO pages. I must say that it was slightly better in an earlier incarnation of the AMO web site, now the link to the latest versions hides shamefully at the bottom of the page, and its labeled misguidingly “View Older Versions”. This is not obvious to the end users. IMHO there is probably more chance of the users looking at the top 3 reviews and finding that their host version is supported or a certain bug is fixed and then telling them where to download the unreviewed version than them preiodically “poking” the versions page to see whether there was a new version available. But you know this already, as you’re trying to cut down the queue – it is just not obvious how it can be done without increasing manpower…

    Also, there is a link for getting the latest published version for each extension, would it be possible (and desirable) to craft one for the latest “experimental” version as well? Or do you think that it would open the door for all sorts of security risks?

  9. iNsuRRecTiON wrote on

    Hi there,

    why not finally improve the add-ons certification system or code it from the scratch!

    I’m, as a user, hate those “(Author not verified)” message in the extension installation popup warning window!

    Why I get exclamation mark warning, even from extension from mozilla devs?

    Here is something very wrong the user friendly GUI.

    If I download and install an extension from AMO, which is reviewed, NOT experimental AND digitally signed with an certify/certificate, I should get a better and more user friendly add-on installation permit window..!

    Like a green checkmark (instead of the yellow exclamation mark!),
    besides the info/warning text says for example, “it’s safe to install this extension, it is reviewed by Mozilla, verified and digitally signed.
    But still beware what you install and use!”

    This is a long standing issue and should be improves very soon (e.g. with the release of Fx 3.7), rework is overdue!

    AND finally I get back to the topic, with that new authentification system, you can accept small code changes/bugfixes faster, for e.g. add-on authors which developes the submitted and changed extension already for some years, like two or three.

    regards,

    iNsuRRecTiON from Germany

  10. Tester wrote on

    Hi there,

    that’s great, keep it up 🙂

    regards,

    Tester

    PS: Does this comment work?!?

  11. Brett Zamir wrote on

    Great work!

  12. Kevin Lee wrote on

    Any data now as to what the wait time is?

    Ways to expedite (I think app developers would even donate to the foundation).

    1. Jorge wrote on

      We post weekly reports at the forum: https://forums.addons.mozilla.org/viewforum.php?f=21