Axel Hecht Mozilla in Your Language

I’ve just uploaded version 0.9 of compare-locales onto pypi. It’s finally the version that does all the fancy value checks that I’ve been talking about for a while, and that some of the localizers have seen flying by in their bugmail.

Here’s what it does:

For DTDs, I create fake xml docs, and try to parse them. This should find encoding errors, as well as unbalanced XML tags or stray ‘&’ ampersands. There’s one thing that’s tricky, and that is references to entities. I do get the list of entities from en-US, so I do have a good idea which should work (really, please). On the other hand, referencing other entities may not be an error. &rdquot; for example could be totally fine. If referenced in an XHTML document, that is. Not if it was included in a XUL document. Of course both breeds could include the same DTD file. I can’t really tell, so I’ve added a new category of reporting, called warnings.

For properties files, I check a bunch of printf tricks. Some of those are warnings, some of those are errors. Which is which basically depended on code-inspection. I also did some heuristics based on comments referencing the plural docs to check for our plurals-special variable handling.

Outstanding are the installer variable checks still, didn’t want to hold back this release for that. They’re somewhat tricky in the details and yet more tedious to get right than the other checks.

What does that mean for localizers? You wanna get the error count down to zero. The warnings count may or may not go down to zero, that’s your call.

The new version isn’t in public use anywhere yet, the deployment will go like this:

• Get a round of public feedback on this release.
• Use on the dashboard (likely gonna happen when I do the 2.0 branch dance, too).
• Try to get the new version used on the build system.

Please give the new version a bit of pounding in your local l10n-merge builds, too. It should strip entities with errors from your localization, and merge in en-US strings for that.

Feel free to file bugs on issues you find.

I’ve been talking with Seth today on how we can answer questions about the status of l10n. My grumpy argument was that I wouldn’t know how to make graphs over time actually show progress, instead of just “failure”. I had two naive graphs, one is showing all missing strings summed up over all locales. That graph would be dominated by the long tail of several dozen locales with a few hundred strings each, and you wouldn’t see a dozen fighting over a few strings each.

The other is what I nick-name “porcupine graph”, show how many locales have no missing strings, vs those that have some missing strings. This is what’s actually implemented on the l10n dashboard as tree progress graphs. But how ever small a string change would be, it goes to all red. And it doesn’t help that one can’t mix green and red color gradients, so the graph usually shows spikes of red and a little black.

Who’d want that as their progress stats, huh?

Now, during the chat with Seth I came up with the idea to just give a little bit of leeway, and accept some missing strings to be OK, at least for some time. I filed bug 582280 on that, and made a rough initial implementation of it. Nothing fancy, just a constant ignored bound of missing strings. Let’s see how the past two weeks of Firefox 4 look now, with just a total of 5 missing strings being OK, ?bound=5:

Now Churchill won over the porcupine, but it’s still pretty red. Which is OK, we haven’t even branched yet, right? So I went ahead and figured I’d add an option hideBad:

Wow, progress. This graph actually looks like our community rocks as much as it does. Gets me grumpy, because this was really just about half an hour of work, plus a few years of thinking.

Now, how do we look on the long run, say, well over half a year? Bumping the bound up to 15, we’re doing like

Pretty good, heh? You can play with it on the dashboard, too. The overall take aways would be:

We have about 20 locales that really track trunk.

We didn’t have that many landings with a high amount of added strings.

I like both :-).

We intend to move from components per locale in the “Mozilla Localizations” product to a matrix of products per locale, and components for each of Firefox, Thunderbird, et al. I’ve created an add-on to set up the products and components and laid out in the newsgroup thread. I wanted to share some screen shots on how things look locally now.

enter_bug.cgi?classification=Mozilla in Your Language looks like this:

Localizers can edit the descriptions on localize.m.o. I’m not totally convinced that the current formatting of the products are great. The double () braces disturb me, both here and on the actual bug form (see below). I might prefer “l10n:ab-CD Language (Region)”.

This is the actual bug entry form, and shows the localized component description. It also shows a rather confusing line wrapping of the product name.

Another aspect that we were concerned about was how it’d look if you changed the product of a bug. Locally, this looks like this now:

Got comments? Please leave them in the original newsgroup thread, or here.

It’s been a while, but I’ll take a stab at what we did to localize Lorentz, i.e., how did we add strings to a stable branch?

The underlying principle was add-on compatibility, that is, language packs that are out there and compatible with 3.6.* needed to continue to work. The second principle was that while we didn’t want to break existing language packs, we did want to offer our localizers to work on the new strings to get the best possible user experience. So we’re adding strings without adding strings.

Given that we had a project branch for Lorentz, the work on the Lorentz branch itself was rather simple, just code like you’d always do. We didn’t do any special branch mechanics there, and built the Lorentz branch against the regular l10n repositories on 1.9.2. The only caveat here was to not remove strings, only additions were OK. As soon as that work got stable in terms of the strings used, we landed a patch that, for each new string, adds a hard-coded fallback to the code. That way, you pick up the localized string without failing if it doesn’t exist. So much for the work on the project branch.

On the stable branch, things are a bit more tricky. Of course we needed to be able to continue shipping security updates from the stable branch, that is, we didn’t want to turn builds red for strings that aren’t really missing. Yet we wanted to signal that the strings are there and to be localized. Up until then, we only had two stages of strings, missing or not. The latter is what we use for handlers in region.properties, for example. We needed a third stage, “report on this string, but don’t fail”. So I added a third stage to the logic around filter.py in compare-locales. This logic is now used in filter.py on the 1.9.2 branch, but not on central. Another necessary addition was to support filter.py for each module, so that we only had to hack around the toolkit strings once, and ont for all apps on 1.9.2. So there’s now a filter.py file for toolkit on 1.9.2, making sure we only report on the lorentz strings in toolkit.

Once that logic was in place, we added a “strings-only” patch to 1.9.2, which had the benefit of exposing the en-US strings in the regular repository that our l10n community is working on.

The outcome today is that out of the 74 languages we ship, only 25 still rely on the en-US fallback strings.

In bullet points, that’d be:

• Do not remove strings, so that newer language packs are compatible with older releases.
• Add strings with en-US fallbacks, so that they can be localized, but don’t error when missing.
• Make filter.py for the module (browser/mail/toolkit) “report” on the new strings, so that localizers see the strings without hiding real errors from non-fatal ones.
• Land strings early on the stable branch to get them exposed to l10n (with filter.py, see above).

The project tree for Lorentz is winding down at various places, and the l10n dashboards make no difference.

The l10n builds off of the projects/firefox-lorentz project tree named “lorentz” have been stopped, and the sign-offs are closed.

Sign-offs for the ongoing work on 3.6.x including the lorentz strings should now go back onto the fx3.6 app version. Sign-offs from the lorentz tree have been migrated to fx3.6 where it made sense to me.

I’ve released a new version of compare-locales, and you should really update.

The new version of compare-locales adds:

• support for more than one filter.py
• support for filter.py returning more than just bools, but “error”, “report”, or “ignore”.

Why?

Lorentz strings. They are missing, but them missing isn’t fatal. So we needed a third state.
And some of them are in toolkit, so we’re moving parts of the filter.py logic from all over the place into
releases/mozilla-1.9.2/toolkit/locales/filter.py. That means that we can remove the hacks in comm-central and mobile-browser, making our life so much more reliable and predictable.

The changes to 1.9.2 will land shortly, so versions of compare-locales prior to 0.7 will stop working.

Update paths:

easy_install -U compare-locales
is the easiest way to do it. Depending on OS and local settings, you might want to
sudo easy_install -U compare-locales

Or,
hg clone http://hg.mozilla.org/build/compare-locales/
and do whatever you did last time, setting paths or python setup.py install.

How does it look? Where’s beef? Or veggies?

Here’s what 0.8 spits out for German on 1.9.2 with patched filter.py:

de
   browser/chrome/browser
     browser.properties
         +crashedpluginsMessage.learnMore
         +crashedpluginsMessage.reloadButton.accesskey
         +crashedpluginsMessage.reloadButton.label
         +crashedpluginsMessage.submitButton.accesskey
         +crashedpluginsMessage.submitButton.label
         +crashedpluginsMessage.title
     preferences/advanced.dtd
         +submitCrashes.accesskey
         +submitCrashes.label
   toolkit/chrome/mozapps/plugins/plugins.dtd
       +reloadPlugin.middle
       +reloadPlugin.post
       +reloadPlugin.pre
       +report.disabled
       +report.failed
       +report.please
       +report.submitted
       +report.submitting
       +report.unavailable
de:
keys: 940
report: 17
unchanged: 634
changed: 4561
87% of entries changed

You can see the regular output of missing strings, and you’ll recognize all of lorentz. New here are three things:

The strings in the long display are not counted as missing, but are in a new summary item called “report”. Those strings are not fatal, but should get localized.
The return value of compare-locales is only dependent on *missing* strings, i.e., code checking the return value will see a successful run of compare-locales if there are reported strings, as long as there are none missing.
If you switch l10n-merge on, it won’t merge the reported strings, but rely on the real code falling back as intended.

Not-so-important feature update, compare-dirs is now supporting l10n-merge, too. That’s sweet for the upcoming weave stuff.

Questions are welcome here, bug reports are welcome in “Mozilla
Localizations”, “Infrastructure” component.