Axel Hecht Mozilla in Your Language

October 26, 2008

New l10n dashboard entry points

Filed under: L10n,Mozilla — Tags: , — Axel Hecht @ 9:13 am

As requested in the l10n sessions in Barcelona, I added locale entry hook. Getting all builds for, say, German, you would go to http://l10n.mozilla.org/dashboard/?locale=de. HTH.

October 21, 2008

l10n-merged linux builds on the l10n server

Filed under: L10n,Mozilla — Tags: , , , — Axel Hecht @ 2:43 pm

I reached another milestone on the l10n builds on the l10n server – reliable l10n depend builds.

A short recap on why they could not be reliable. Details are in Armen’s and John’s presentation in Whistler. First and foremost, l10n builds with missing strings break. They might start, or not, maybe even crash. Or just display the yellow screen of xml parsing death. Now, l10n builds are not really builds, but repackages of an en-US build. Between the time that the en-US build started, or, in hg, the revision it used, and the tip at the time when the en-US binary is finished and available, there can be further l10n-impact landings. We are using the nightly builds for the repackages throughout the whole day even, so the chance that the current en-US source doesn’t correspond to the nightly increases. So even if you know that a localization is good tip vs tip, you can’t say if it’s breaking the previous nightly or not. Huh? Look at the graphs in Armen’s and John’s presentation for more arrows going back and forth in time. ;-)

Enter bugs 452426 and 458014. 452426 added the changeset id to application.ini (thanks Ted), and 458014 refactored browser/locales/Makefile.in with additional logic to extract that info for the build system. I got that one landed yesterday, so we can now get the source stamp of mozilla-central for a firefox build.

Right, good catch, this doesn’t work for comm-central builds. I’ll leave it up to them to figure out how to reproduce the plethora of repos they have.

So far, so good. You download the nightly, unpack, ident (the rule to extract the changeset id). Now you back to your source tree and hg update to that revision, and run compare-locales against that. We’d be able to reliably say “works” or “better don’t touch”.

We promised more, and more pieces came together today.

With reliable compare-locales code, one can not only detect missing strings, but also add missing strings to files. Think about a CPP step, nothing permanent, nothing gets landed upstream. But just for the needs of this particular build, you’d have something that has all strings. Not all translated, some padded from en-US. That works. compare-locales is already able to do merges for a while now, storing the changed files into a separate location. Mostly because I consider changing the source to be evil. So what about missing files? Nothing. Good files? Nothing. How does the build pick up files from merges, l10n, and en-US then?

By rewriting make-jars.pl, enter JarMaker.py. Among overall readibility improvements and removing XPFE hacks, JarMaker.py offers to pick up l10n files from a list of top-level source dirs.  It offers another cute feature, by writing both chrome and extension manifests at once. Now, with bug 458014, we don’t have to run the libs phase for installers and langpack separately. (I never got why we do that until I rewrote make-jars.pl, actually.) The rewrite of JarMaker.py was preceeded by rewriting Preprocessor.py, so that all of the jar generation can happen in a single python process.

Starting from today, all of this came together with my installation of buildbot on the l10n server.

This gives us

  • builds on push, i.e., feedback within 5-10 minutes (real stats pending)
  • comparisons of the l10n tip against both
    • the en-US tip (for the upcoming nightly)
    • changeset of the previous (for the existing nightly, with l10n-merge)
  • html-ified output for both of those
  • updates for the dashboard

and last, but not at all least, a

  • working build, even for partial translations.

Find 60 3.1b2pre linux builds on the l10n server.

Thanks to Armen, I used a few of his new makefile targets for download and upload, he did a bunch of work for the sourcestamps-in-application.ini on cvs, too. Thanks to Ted, the poor fellow had to review all my rewrites and Makefile dependencies foo, and did some patches, too. hgpoller stuff not to forget.

TODO:

  • silme will offer even more reliable merges
  • nightly scheduler for all locales (currently I only build on l10n and en-US l10n-impact changes) (*)
  • mar’s
  • comm-central
  • more Makefile foo to pick up more missing files from en-US in doubt… (*)
  • … or at least document the core set of required files (*)

I won’t take most of those, fwiw. Possibly only the (*) ones.

Sources are in my tooling repository, and there’s an updated version of compare-locales, 0.6, on pypi. No drastic changes here, just some paths fixes, mostly for Windows.

October 2, 2008

State of the Zwiebelfisch

Filed under: L10n,Mozilla — Tags: , — Axel Hecht @ 3:45 pm

For those that know that I’ve been somewhat offline lately, good news. I’m back online. I’m not going into the details, but ISPs in Germany and their handling of customers is, errr, let’s put it that way: not Firefox.

Other items of progress lately:

Seth covered Firefox 3.1 Beta 1. In my words, it went like I expected, our localizers just blew our expectations. That doesn’t make me change my expectations. I expect what I think we should expect, and heartly welcome that Firefox gets so much more. 35 localization teams swallowed the even more technical hg compared to CVS. Some localization teams volunteered to help out others. Overall, the current stats are that we have 41 hg l10n repos with localizers commiting, 21 with just me so far, and 6 are plain empty. The last 6 are waiting on good content on cvs to migrate over to hg.

I just landed changes to both hgpoller and hg_templates that add a json output to hg.m.o, once that’s pushed live. That json output is designed to allow to clone the pushlog2.db, i.e., with a mere clone of a repository on hg.m.o and a copy of pushlog2.db, you can do all kinds of history magic. I sure hope to use that data to make out opt-in process a lot more appealing and understandable.

The real big thing in the queue is bug 458014, refactoring browser/locales/Makefile.in. There are a bunch of things I’m doing there. Not wasting time and computing power would be one. Being able to actually tell which build the build is repackaging before it is repacking is the other big thing. The initial comment in the bug has a good analysis.

This is a rather big step for l10n in mozilla land. Being able to actually pick the right source in en-US for the build you’re repackaging is something we’re talking about for years. It’s a prerequisite for doing l10n-merge at build time, too. Which that bug adds, btw. So this bug should really pave the path to reliable l10n nightly builds, good testing infrastructure, and more.

“More” being a new way to start localizing Mozilla applications. No more, no less. Up to now, your localized build was busted if the localized source you used to build wouldn’t include all the strings of en-US. With the changes I added in place, you can just make the build insert those at build time. This will not change the output of compare-locales, and will give localizers the ability to, without additional tools, decide on whether to keep a particular string with the same value as en-US or to bother about it later. l10n-merge is really only going to add an intermediate output. Think about it as CPP output.

We talked about this l10n-merge in Whistler, and there are a bunch of loose ends and room of improvement. Nevertheless, the state in bug 458014 is such that I would like to use to figure out how to make a buildbot factory do the right thing at least in the more or less trivial cases.

l10n-merge won’t change our policy to only ship complete localizations. I hear folks trying to change it, thus the comment. It’ll be tough to convince me to not require complete localizations. “Localizations” doesn’t imply translating all strings, though. Localization implies an conscious decision on whether a particular string should be translated or not. Security error messages, of course, those protect our users on the web. XSLT error message much more likely not, there are only a few languages which have a community around XSLT in their native language. Falling back to English without a consious decision is a door wide open to a sucky user experience. Something that at least I don’t consider to be Firefox. Maybe something else. Minefield is one possible name for it, there might be others.

PS: I picked Zwiebelfisch for no other reason than “it’s late at night, and it starts with Onion”

Powered by WordPress