Mozilla Add-ons Community Blog
Categories: documentation end users

How we manage user review moderation

6 responses

Anyone with an AMO account can leave a user review and rating for an add-on. User reviews are an invaluable part of AMO, helping users decide which add-ons are the best at what they do. However, as any open forum on the web, it needs some monitoring in order to keep the content relevant and useful. We have a set of writing policies and a moderation process for handling user reviews. A link to the policies is presented when you post a user review, but the moderation process may not be very clear, so I’ll explain it here.

We don’t actively monitor users review posted on AMO. Instead, we provide a way for users to request moderation for reviews that are in violation of the content guidelines. Anyone with an AMO account can do this, including the developer of the add-on for which the review was posted. You can find the moderation report link by clicking “See all user reviews” on any add-on listing page and looking at the bottom-right corner of a review. It’ll let you choose the reason for the report and add some information to it.

After a user review is reported, it goes to a moderation queue, where a member of our add-on review team will give it a look and decide if it should stay or be deleted. This is the same team that handles add-on code analysis (confusingly also called add-on reviews). It is mostly comprised of volunteers with a background  in add-on development.

Policy

We want reviews to be useful for add-on users, so our policy encourages user reviews that speak to the advertised features of the add-on. User reviews aren’t meant to start a conversation with the developer or other reviewers, nor are they meant to act as a bug reporting tool. Most add-on listings on the site include contact or support information to get in touch with the developer about these issues.

Deciding what is or isn’t useful is a very subjective call, so it is difficult to obtain results that are consistent, or satisfactory for all users. Some people will get annoyed if we delete a user review that is critical of an add-on, and proceed to write new reviews complaining about the deletion, which are then also deleted. It’s a sub-standard experience that we are aware of and want to fix.

Limitations

While user reviews and their moderation are an important part of AMO, they haven’t had much work done in many years (I should note this isn’t the only part of AMO that has this problem). We don’t have a good way to undelete reviews, or to communicate the reason behind a review deletion to its author.

We haven’t changed our review policies for a long time, and they need some work. Moderating reviews is a secondary task for our review team, and having a team composed of add-on developers probably introduces some bias against negative user reviews. We have been thinking about creating a new team in charge of user review moderation, ideally with less of a developer mindset. I think we’ll also need to relax our policies in order to encourage more diversity of opinions.

User reviews aren’t abundant, and we only show the last 3 in the add-on page. This means that an unwarranted negative review can linger for a long time, causing the developer to lose users. We need to implement a way to mark user reviews as useful or not so we can surface the most useful ones (a common feature found on sites like Amazon and the iTunes Store).

Unfortunately our main limitations are time and people, so it might take a while before we can change user reviews in a meaningful way. Separating the review team and changing the policies is something we can do relatively quickly, so I hope we will be in a better place early next year.

6 comments on “How we manage user review moderation”

  1. Ben Bucksch wrote on

    I like the current policy, esp. the “no bug reports” and “review speaks to the advertized features” parts.

    There’s unfortunately a set of people who are both mindless and vocal, and they don’t read the addon’s description, they are surprised about an aspect (which they could have expected, if they had read the description), and then trash the addon out of frustration.

    As you rightly point out, these unjustified reviews often linger around for months, and drag down the average notably, unfairly disparaging addon. For that reason, I think the current moderation of reviews is good.

    Some suggestions:
    * Make a moderation team that’s separate from addon reviewers. It’s a completely different skillset, and you should be able to find moderators more easily.
    * Give a way to submit bug reports, in the same form (this part is important!) that reviews are submitted. They’ll be sent to the addon author by email, optionally to a dedicated email address, allowing them to be filtered or automatically entered into a bug tracker or ticket system on the author’s side. — This should take care of the “bug report” reviews.
    * Actively encourage users to rate addons (without (!) review text, just the star rating), to get a better average.

    1. Aris wrote on

      I agree with your opinion and suggestions except the last one.
      “Actively encourage users to rate add-ons (without (!) review text, just the star rating), to get a better average.”
      Frustrated users, that are unwilling to read descriptions and to use support offers (forums, email etc.) or that simply misunderstood what the add-on is for or is capable of, could leave bad ratings without any way for devs or review moderators to decide whether such reviews (without review text) are valid and correct or not. This would do more harm than good to AMO and only frustrate devs.

      In my opinion it would be a good idea to add a confirmation box before posting a user review. Something like
      “Are you sure your comment followed our guidelines and does not contain any bug reports or insults out of frustration?” would probably make some users to rethink what they have written.
      In addition user reviews should not appear immediately, but after lets say 30 minutes giving users the opportunity to edit their reviews in that time without anybody to notice.

  2. Axel Grude wrote on

    I think an open voting system for user reviews would be great. I have some addons that have almost exclusively 5 star reviews and then there is one recent 3star at the top that I really do not care to delete as my answer to it might help other users. However it does tell the wrong story that one third of my reviews are bad when it is one out of a hundred. If reviews could be upvoted by users then the most useful ones could bubble to the top. more to the point if we allow downvoting as well, we could simply stop worrying about less useless reviews of the nature “doesn’t work for me”.

  3. Udo wrote on

    > I think an open voting system for user reviews would be great.

    No. It would have the problems as voting with text, but stronger. E.g. if the reviewer discussed only one aspect of an Addon, maybe an exclusion criterion, then it would rated bad more easily. Thats my experience with consumer review sites.

  4. onemen wrote on

    The main problem with the current AMO design is that the link to the developer site and mail are almost invisible among the clatter of information button and links.

    I thing that many users do believe the review button is also for bug report.

    The main problem for me, as a developer, is the lake of ways to communicate with the user that issue the bug report in order to properly managing the process of fixing bugs.

    I thing that adding a button to report bug near the “Write a review” button is a must !!!

    The best would be to add form/site dedicate for add-on bugs (using Bugzilla ??).
    The steps to report a bug must be simple, normal users can not deal with forms like in Bugzilla.
    It is important to simplify the ways users can post information to the developer with the bug report about the operating system, Firefox version, extension version, other extensions installed, theme, preference etc…

  5. Ludo wrote on

    Although sour comments must be useful to users, but I thought that we should not too severely moderate comments expressing strong views on the operation of an addon. Indeed if a user has problems with an addon (if it works) is that it can be poorly explained. This negative feedback can help the developer to produce a more complete and better explained documentation.