Anyone with an AMO account can leave a user review and rating for an add-on. User reviews are an invaluable part of AMO, helping users decide which add-ons are the best at what they do. However, as any open forum on the web, it needs some monitoring in order to keep the content relevant and useful. We have a set of writing policies and a moderation process for handling user reviews. A link to the policies is presented when you post a user review, but the moderation process may not be very clear, so I’ll explain it here.
We don’t actively monitor users review posted on AMO. Instead, we provide a way for users to request moderation for reviews that are in violation of the content guidelines. Anyone with an AMO account can do this, including the developer of the add-on for which the review was posted. You can find the moderation report link by clicking “See all user reviews” on any add-on listing page and looking at the bottom-right corner of a review. It’ll let you choose the reason for the report and add some information to it.
After a user review is reported, it goes to a moderation queue, where a member of our add-on review team will give it a look and decide if it should stay or be deleted. This is the same team that handles add-on code analysis (confusingly also called add-on reviews). It is mostly comprised of volunteers with a background in add-on development.
We want reviews to be useful for add-on users, so our policy encourages user reviews that speak to the advertised features of the add-on. User reviews aren’t meant to start a conversation with the developer or other reviewers, nor are they meant to act as a bug reporting tool. Most add-on listings on the site include contact or support information to get in touch with the developer about these issues.
Deciding what is or isn’t useful is a very subjective call, so it is difficult to obtain results that are consistent, or satisfactory for all users. Some people will get annoyed if we delete a user review that is critical of an add-on, and proceed to write new reviews complaining about the deletion, which are then also deleted. It’s a sub-standard experience that we are aware of and want to fix.
While user reviews and their moderation are an important part of AMO, they haven’t had much work done in many years (I should note this isn’t the only part of AMO that has this problem). We don’t have a good way to undelete reviews, or to communicate the reason behind a review deletion to its author.
We haven’t changed our review policies for a long time, and they need some work. Moderating reviews is a secondary task for our review team, and having a team composed of add-on developers probably introduces some bias against negative user reviews. We have been thinking about creating a new team in charge of user review moderation, ideally with less of a developer mindset. I think we’ll also need to relax our policies in order to encourage more diversity of opinions.
User reviews aren’t abundant, and we only show the last 3 in the add-on page. This means that an unwarranted negative review can linger for a long time, causing the developer to lose users. We need to implement a way to mark user reviews as useful or not so we can surface the most useful ones (a common feature found on sites like Amazon and the iTunes Store).
Unfortunately our main limitations are time and people, so it might take a while before we can change user reviews in a meaningful way. Separating the review team and changing the policies is something we can do relatively quickly, so I hope we will be in a better place early next year.