all pre-existing add-ons restored (made public)



Short version: All add-ons that were public in AMO prior to Remora’s launch this weekend should now be publicly available on AMO again. We will be in contact with all add-on authors to provide more detail and clarity on what we’ll be doing going forward to help make the AMO and add-ons experience more polished and accessible for all our users.

Not quite as short version: I made a mistake in choosing how to roll out the sandbox and increased quality standards for AMO, in concert with the Remora software update, and we’re now correcting it. We still believe that the combination of the sandbox system (to provide more transparency and participation in evaluating add-ons before they’re published) and a higher standard for inclusion on the site (especially in terms of polish and author responsiveness) will benefit users and developers alike. But it’s clear from community feedback — often, but not always, quite reasoned and polite — that we didn’t do a good job of communicating what was going on, and as a result both users and developers were hurt by the change.

We’ve now updated the database to make all the pre-existing add-ons public, and we’ll be working with editors and users and developers to make the global improvements that we seek. The new tools and capabilities in Remora should make it much easier to serve the growing add-ons community, and we’re looking forward to even more such improvements in the future.

Thank you to everyone for your feedback, and passion, about AMO and how it affects you. Please know that we share it, and that we want to work with you to make AMO even better.

(In related news, the site performance has been improved dramatically over the weekend, but I’ll let morgamic tell the rest of that story!)

Reviewing, the Sandbox, and “missing” add-ons



Though Justin’s first post about the sandbox was several months ago, there is still a fair bit of confusion about the whats and wherefores of the new system. I’ll try to give a relatively concise description of the goals and current operation of the sandbox system here, but please keep in mind that we are rolling out the first version here, and not the last. We have lengthy and detailed discussions about the feedback provided by people even in its pre-release state, and we are grateful for the suggestions and information that people have contributed already.

The goals of the sandbox system, distilled down to the shortest version I can manage, are:

  1. Assist developers and site administrators in making sure that the users of AMO have great experiences with great add-ons.
  2. Expand the review system to allow participation from a much wider cross-section of the Mozilla community.

And the operation of the current review system is described at a shiny new policy page, including a long-long-long-awaited description of how add-ons can become recommended.

It is very, very important to us that unsuspecting users are protected from malicious or careless software when using the AMO site. The previous AMO version(s) relied on a system of roughly-sequential closed-door review by a small group of volunteers to validate the add-ons that were submitted to the site. As the number of add-ons grew, it became extremely difficult for that group of people to keep up with the incoming requests, and the number of specialized (and difficult-to-review) add-ons grew as well. When Firefox had 2 million users, an add-on that was relevant to 1% of those users could be reasonably left to wait a fairly long time for review, but such a “one percenter” add-on is now interested to almost a million people around the world!

Historically, Mozilla has been able to scale to the width of the web by allowing more members of our community to contribute, be it to testing, coding, marketing or dressing up in costumes and walking around downtown. We wanted to bring this increased scale to reviewing and testing add-ons to ensure that they were safe and well-described for our users.

Additionally, as Firefox’s appeal widens to a more mainstream, less computer-savvy audience, we came to realize that we needed to make sure that the add-ons were comprehensible and accessible to people who were not as comfortable working around installation problems or figuring out what an extension did by experimentation. The add-ons that people find through their use of AMO should be inspiring examples of what the extensible web can provide, and should make users’ lives better in every way. We know that the Mozilla community can provide these great experiences, and we want to both incent and reward add-on developers who provide them. We know that the AMO site is an extremely valuable channel for software to reach new users, and we want to use that to promote and strengthen the values that Mozilla stands for. We are raising the bar with Remora for what it takes to get in front of nearly a hundred million of the most savvy, influential and good-looking browser connoisseurs in the world, and we will continue to raise the bar over time as we feel it is in the interests of Mozilla and, most importantly, its users.

So we now have the sandbox, and as you’ve already read it’s where add-ons start their lives. We chose a threshold to start with to seed the public site, and we expect and hope that users of add-ons that they feel belong in the public side of AMO will write informative reviews and help the AMO editors find those gems that we — I, if you want to point a finger more closely — didn’t have in the public site on day 1.

Over time, I expect that we will make the sandbox more visible to new users, as we learn more about how to balance the need to protect unwitting users from add-ons that have not been tested with the desire to match more users up with “niche” add-ons or up-and-coming experiments. We’re already looking, based on just the pre-release feedback, at making direct links to non-public add-ons work with appropriate caveats and warnings. (They would still be hidden from search and browsing.)

I hope this helps to explain what we are trying to achieve with the sandbox system, how the system works, and how you can help make it work better on small or large scale. We’re still hard at work making sure that Remora is good and ready before we put it live, and we know that the first little while after launch will bring the need for refinement in how the system works, but we believe that these changes will work to strengthen the add-ons community and ecosystem, for the benefit of users and developers alike. That is, after all, what we’re all about.

Remora update and plan



As morgamic wrote on Friday, the Remora team have all had a long 9 days trying to get the new version of AMO out the door and make all its improvements available to our users. As we discovered to our chagrin, our performance testing on the stage environment did not really map well into real-world results, and the result was unacceptable load on the app cluster — unacceptable to the extent that pretty much everything that was sharing that database was killed by it, in fact.

With the assistance of IT (and especially of oremj, who wrote and operated a load replay tool to help us make sure we were testing the right mix of requests), we’ve been able to gather more data on where we were too slow, and since Friday we’ve been able to make some rather significant improvements to our impact on the database. Specifically, if I may be permitted a nerdy digression: on Friday, a single application server was able to generate a load of 7.0 on a single database server, but in our tests today, that same application was only able to generate a load of 0.04 (sic). We have more performance options still in our quiver if we need them, but because they are all non-trivial in terms of risk of regression we are hoping that we’ll have reached an acceptable level with this most recent work and will be able to hold off on additional optimizations until after the initial release.

Over the course of the week, and even the last few days, we’ve seen that the performance improvements we made did indeed cause us to have regressions, and while our test suite was able to catch some of them, it did not catch all of them. We want to get more user (including add-on developer) testing to help us prove out the system, and also to collect more of the great feedback that’s already been so helpful in improving the site. Also, we have learned all too well that running tests on our stage setup is not a surefire predictor of performance on the production cluster.

And if that weren’t enough, we now have a snapshot of the existing AMO database that’s more than a week old, and is missing many changes since that time. We don’t want to casually throw away the work of our community or reviewers, so we are faced with the need to re-migrate the database — and incorporate changes that people have made on preview.amo to update their add-ons for the new capabilities of Remora as well.

So what’s next?

Measure twice, cut once

First, we have a maintenance window with IT right now in which we will not be deploying Remora, but running full-scale load testing against the preview installation. This should give us a pretty reliable picture of how Remora will perform “in the wild”, and will give us additional data on where our remaining hot spots are. If these results show that we’re still substantially worse than the current AMO in terms of impact on the cluster, we’ll have to circle back and pick another appropriate performance arrow to shoot. We are optimistic, though, that our 17,400% improvement will put us in a pretty good position here.

Retrace our steps

Once we’re confident that we’re in scoring range in terms of performance, we’re going to lock down the current AMO system such that new add-ons and updates can’t be submitted, and we’ll direct developers to the new site for submission of new add-ons and updates. This will introduce perhaps a few weeks of lag into the update process, but it turns out (alas) that the current system’s review-scaling problems are such that updates sometimes have to wait for at least that long today, so it’s not as shocking a situation as it might be if we were coming from a more efficient system. For high-priority updates (security updates, or updates to our most popular add-ons), we’ll be instructing people to file bugs, and we’ll handle the updates by hand on the current site. It’ll be a bit painful, but we don’t want to find ourselves in this re-migration situation again before we release, which would be even more painful.

Tell us what you really think

We’ll also be driving more users from the current site and other parts of the community to the new site, to get their feedback and to get developers to work on updating their categorization and summaries, for example. This should help us catch a lot of simple polish bugs, as well as improving our general confidence in the app significantly.

Make a list, check it twice

Based on their feedback and our own additional testing over the next week and a half (again, assuming that the performance groundhog sees his shadow tonight), we’ll construct a release-blocker list, and from that an end-game release schedule.

What’s this all about, anyway?

And then we’ll have Remora live as This will give us some powerful tools for bringing more of the Mozilla community into the add-ons world, including:

  • deep support for localization, including translation of add-on descriptions, version notes, and user reviews
  • a more inclusive and transparent review and nomination model for making sure that the add-ons that we put in front of 80M users are up to the challenge, and that users get what they expect with clear descriptions and meaningful ratings
  • a discussion system for users and developers to communicate with each other, helping to provide a better channel for feedback and support
  • a more robust code base for even more improvements in the future

Fool me once, shame on…don’t be fooled again

Or: “didn’t you tell a bunch of reporters that you were releasing on February 12th?”

It’s embarrassing and frustrating to miss deadlines (as I know from very extensive experience!), and painful to not be able to put your hard work out into the world where it can help people. The entire Remora team has been working almost literally around the clock for the last few weeks, and very hard for half a year before that, and we would like nothing in the world more than to deliver it to the world.

Well, one thing in the world that we’d like more: to be confident when we release it that it’s up to the daunting task of representing add-ons to many, many millions of Firefox users, and that it’ll be a release that we’re appropriately proud of. We’d rather be late than damage the world’s impression of Mozilla or add-ons, and we’d be doing a disservice to the exact users we’re working to help if we were to push it out before we were confident it was ready.

It’s been a tremendous learning experience building an application with such a huge exposure to the world, and a rewarding one, though not one that we would necessarily want to do every quarter. We’re ready to correct our course and move confidently towards the finish line, or perhaps towards another metaphor of completion entirely.

Thanks to everyone for their patience and support, and especially:

  • to the IT team for their support and creativity in helping us find a path to better performance,
  • to our community of testers (I’m looking at you, Wladimir) for not only helping us identify problems but also suggesting solutions to them,
  • to our localizers, for their patience with our early disorganization and frequent string changes.
  • For the (somewhat exhausted but still pretty upbeat) Remora team,

    Mike Shaver

AMO v3 (Remora) launch delayed 24 hours



As I’m sure many of you are aware, today (Feb 12th) was the scheduled release day for Remora, which will bring localization, a more inclusive review system, and discussion capabilities to the site. We had a maintenance window scheduled for today from 7-11pm Pacific, and during that period we discovered a number of issues related to differences between our test environment and the production one, as well as some late-breaking bugs found by our testers.

While we were able to resolve all but one of those issues to our satisfaction — and that of our test suite — we ended up making sufficient changes to the system that we wanted to take another day to test it in the production environment before we put it in front of Firefox’s tens of millions of users.

It’s a heart-breaking decision for the Remora team, but we think it’s the right thing for our users, and we’d rather have 24 hours of disappointment than put the users’ experience at risk.

On the bright side, this will give us an extended window to take updates from our intrepid localizers, as those changes are sufficiently low risk that we’re comfortable taking them during tomorrow’s test-and-test-some-more window.

Thanks to everyone for their patience and support; we’re so close now that we can taste it.

Localizing AMO3 (part 1: overview and static)



As morgamic mentioned, we’re very excited about finally being able to provide a great add-ons experience in more than just English. (Don’t get me wrong, I love English, but there are other great languages as well.)

There are three major parts to localization in the “Remora” code base:

  • Localization of static web content. These are the sorts of things that you would find hard-coded into markup in the current site, and includes the text of links to different sections of the site, error messages, and help text. We’re using gettext for this, and performing the translation should hopefully be pretty straightforward.
  • Localization of dynamic site content. Right now, this is pretty much only the names and descriptions of categories, but will likely grow a bit to include other kinds of database-driven site-wide text that users see. (There are a few more examples right now on the admin and reviewer side as well.) We don’t have good tools for this today, but there are few enough entries in need of this sort of translation that we can probably manage with manual entry by an admin for a while.
  • Localization of user and developer content. Add-on descriptions, screen shots and their captions, review text, version notes, that sort of thing. This localized data is managed by the site users themselves, so localizers shouldn’t need to do anything special to accomodate it. (It’s possible, perhaps even likely, that localizers will want to help authors get localized descriptions of their add-ons, but that’s another discussion entirely.)

This post is about the process for doing the first part of the localization (static gettext translation) of Remora. Hopefully it’ll be quite straightforward, especially for people who already have experience with gettext-based localization. There are currently a little fewer than 500 strings to localize, but only about 300 of them affect the public site.

(If you’re the sort of person who can set up a moderately-complex PHP application, then you might want to follow the installation instructions to set up your very own Remora, but that isn’t strictly necessary for localizing the site.)

First, you should learn a language other than English. We can’t provide assistance with this step, unfortunately.

With that out of the way, you’ll want to get a copy of the English gettext “po” file. You can find the trunk version of it here. That file will be updated live with our development, so we might end up wanting to have localizers follow tags, but for now we want to let people stay as up-to-date as they’re willing, and we don’t expect a ton of changes to the public-facing strings before we deploy Remora.

You can translate that file by hand, or there are various tools to make it easier. Wikipedia has some details here, though you won’t have to compile things to “mo” format.

Note: this file contains all the strings for the site, including those for the admin/reviewer/developer parts. The strings for those parts of the site are not frozen yet, so translating them at this point will likely be a bit of a waste, since they’ll need to be revisited in the nearish future. We recommend that translators only bother with msgids that are a “key”, like “user_form_firstname”, and skip the ones that use English text as their msgids, like “Back to Review”. As strings become more frozen, they’ll be converted to use the key form.

Once you have a translation complete, you should take the .po file and attach it to a bug filed against the Public Pages component of The summary of your bug should be of the form “gettext for AMO (your-locale-here)” — please look for an existing bug beforehand, ideally before starting your work! If you haven’t been active in localization of Mozilla products before, you should probably try to connect with the existing localization community for your locale of choice, to avoid duplication of effort and to help you get up to speed.

(We’re going to be working with the localization leads to figure out how to best manage “module ownership” of different site localizations, so this process may change in the near future. We appreciate your patience in this matter!)

So, to summarize:

  1. Check to make sure that nobody has already done the localization that you’re interested in!
  2. Get the English string file to start from.
  3. Translate the msgids with names_like_this into your language.
  4. Attach the resulting “messages.po” file to a new bug.

As implied by the title of this post, there will be other posts to come on localizing other elements of AMO, but the static part should give people a great start on giving us an add-ons site that better reflects the global nature of the Mozilla community.