Add-ons Update – Week of 2014/04/16

Peter wrote on April 16, 2014 at 1:48 pm:

Had to wait in the queue for 3 weeks (!). Meanwhile a Chrome extension is distributed within 30 minutes. Maybe you should look into automating the review process…

1. Jorge Villalobos wrote on April 19, 2014 at 3:12 am:

   Automating the review process makes the system very insecure, so that’s not an option for us. It’s unfortunate that we’re a bit lacking in volunteer support at the moment, but we thinks it’s the best way to ensure a safe experience to our users.

   1. Luke wrote on April 19, 2014 at 5:41 am:

      Glad to have a real person reviewing these, it keeps quality up and prevents spyware/adware apps and renamed ripoffs of common open-source apps, as we see in Google Play.

      By the way, any results yet from the Firefox 29 redesign contest that ended this week?

      1. Jorge Villalobos wrote on April 19, 2014 at 8:16 am:

         We’re only starting with the judging period. It’ll take a couple of weeks before the results are published.

   2. Peter wrote on April 20, 2014 at 8:34 am:

      Well, human reviewers are not exactly more secure. For example one of your reviewers approved an update of my add-on. In the next update, which only included some minor tweaks, another reviewer denied the update for a reason the first reviewer approved. Because of this, and the waiting period of 3 weeks between the reviews, several thousand users have an add-on installed that was labeled as insecure by Mozilla.

      Of course there is no actual security problem in the add-on. The reviewer criticized the lack of escaping two variables. However those variables are set by the add-on (not the user) and can’t cause problems. But as the second reviewer put it: they don’t have time to look at changes in context.

      In the end I agreed to escape the variables and pushed another update. At the current speed of the queue I wonder if 3 weeks of waiting will be enough. On many days it doesn’t move at all. To remind you: in the meantime all users are using the old version which was first labeled secure by the first reviewer and then insecure by the second reviewer. And of course the users miss out on the new features, some of them being support for Australis.

      For some add-ons this slow and cumbersome process might work. For others it doesn’t. I can see why some devs decided to self-host their add-ons.

      If you keep simply denying the need for (at least partial) automation I can only shake my head. Especially with the Add-on SDK you created a platform that can be used as a starting point.

      1. Jorge Villalobos wrote on April 22, 2014 at 2:58 pm:

         There is some automation in the preliminary validation we perform. However, we can’t leave the manual review portion aside, since there are plenty of problems that are either very difficult or impossible to detect automatically. The manual process certainly isn’t perfect, since there’s clearly a human element at play, but it’s better than nothing, which is the only viable alternative at present.

         We’re working on reducing waiting times. Hopefully they’ll be back to more acceptable numbers soon.

   3. S Antony wrote on April 24, 2014 at 11:08 am:

      I second what Peter said. And there must be some process to speed up add-on review times.
      I had a similar experience. After nearly 20 days, my add-on was only granted prelim-review due to a style sheet loading convention (I was supposed to escape the css with -moz-document despite having unique ids for all the elements). I immediately corrected and re-submitted. But I don’t think it will get reviewed any time in another 2 weeks, as the queue is quite big now.

      1. If there are minor problems like these, (or as in the case of Peter above), I think the rules could be relaxed a bit. Unless, it is a critical problem, the add-on could pass the review and the author can be given comments to correct and re-submit. Is there a reviewer’s dashboard that can track such things ?
      2. There must be some way to indicate that updates to a reviewed add-on is only a minor one. I had this experience when the only change was one line of json string, and still I had to wait around 2 weeks. Yes, a ‘minor-update’ switch could get abused, but doesn’t reviewers do a diff with the previous versions ?
      3. I guess many people would happily volunteer to test other’s add-ons so as to get their own add-on reviewed quickly. Although not everyone can review, testing with a pre-defined criterion should still be possible.

      My 2 cents.

      At the end, I understand that reviewing an add-on is a really time consuming process, and I thank all the volunteer add-on reviewers out there!

      1. Simon wrote on April 28, 2014 at 8:05 am:

         I agree with the sentiment of some of these comments, especially Peter and Anthony.

         I sincerely don’t believe some further automation couldn’t help a bit, at least to triage slightly, but i’m willing to accept that could be just a reflection of my own ignorance.

         Further, making small changes shouldn’t lose ones place in the queue. And making changes as per the reviewer requests shouldn’t cause you to go the end of the queue, though i’m no where close to even getting a preliminary review yet.

         1. Jorge Villalobos wrote on April 28, 2014 at 3:41 pm:

            Yes, we agree that submitting new versions while in the queue shouldn’t set you back. There’s a bug opened to fix that. Unfortunately, development on AMO is fairly stalled at the moment, so we need to prioritize very aggressively.