Breakthroughs in the add-on review queues

The last few months have been dense with big changes in the add-ons world. Just reading this blog and looking at all the new faces and posts should give you a good idea. Today I’d like to show you how these changes have affected the add-on review queues and what you can expect for the future.

These charts show the state of the review queues on a weekly basis, starting on Q4 last year:

The green area represents add-ons that have been waiting under 5 days, yellow between 5 and 10 days, and red over 10 days.

As you can see, we were doing pretty poorly in late 2015. We only had one admin reviewer, Andreas Wagner, whose attention was almost entirely focused on reviewing unlisted add-ons (not included in the charts). We already had a significant backlog of add-ons awaiting admin review, and during this time it got a lot worse. We were looking for new admin reviewers we could get on board as contractors, but most of our attention was focused on extension signing.

Turning the tide

A few things happened that helped us get the review queues under control:

• We stopped gating unlisted add-on signing by review. This freed Andreas’s time so he could go back to review listed add-ons.
• Our team (finally!) grew. We now have Philipp Kewisch as our second full time admin reviewer and Noitidart helping part time also as an admin. Other areas of the team also got help, like the AMO dev team and editorial, all of which has afforded us much-needed breathing room and focus.
• Our volunteer team also ramped up their efforts and kept up with the increased submission rate. During the first few weeks of the year, they averaged over 500 reviews per week! This is an amazing feat, especially to those of us who know about the pains of code review.

The review queues are now almost entirely in the green, which is fantastic (in fact, we’re all green in today’s report). However, it’s reasonable to have some skepticism, since we’ve been here before. We rely on a couple of very active volunteers who do the bulk of the reviews. And we will need to turn more of our attention to unlisted submissions soon (though not to the same extent as before).

Keeping it that way

So, what are we doing to maintain wait times that developers can rely on?

First, we have the staff additions I mentioned earlier. Having a stable admin team contributes greatly because we have a higher baseline even when volunteer activity drops. We also have another part-time opening that we will try to fill soon.

We’ve made some changes to the reviewer tools to encourage more participation from less-active reviewers. The review team has grown significantly in the past couple of months, and we receive new applications all the time. Are you interested in joining us? Please apply!

Finally, I think we’re past the big wave of submissions that happened due to the introduction of extension signing. But we do have a stable and steady increase in developer activity (which is good!).

We’d like to thank the add-on developer community for their immense patience during this time. We know it’s been rough and we’re very grateful that you stuck with us. We’ll keep working hard to make sure this becomes the standard.