l10n-merged linux builds on the l10n server

I reached another milestone on the l10n builds on the l10n server – reliable l10n depend builds.

A short recap on why they could not be reliable. Details are in Armen’s and John’s presentation in Whistler. First and foremost, l10n builds with missing strings break. They might start, or not, maybe even crash. Or just display the yellow screen of xml parsing death. Now, l10n builds are not really builds, but repackages of an en-US build. Between the time that the en-US build started, or, in hg, the revision it used, and the tip at the time when the en-US binary is finished and available, there can be further l10n-impact landings. We are using the nightly builds for the repackages throughout the whole day even, so the chance that the current en-US source doesn’t correspond to the nightly increases. So even if you know that a localization is good tip vs tip, you can’t say if it’s breaking the previous nightly or not. Huh? Look at the graphs in Armen’s and John’s presentation for more arrows going back and forth in time. ;-)

Enter bugs 452426 and 458014. 452426 added the changeset id to application.ini (thanks Ted), and 458014 refactored browser/locales/Makefile.in with additional logic to extract that info for the build system. I got that one landed yesterday, so we can now get the source stamp of mozilla-central for a firefox build.

Right, good catch, this doesn’t work for comm-central builds. I’ll leave it up to them to figure out how to reproduce the plethora of repos they have.

So far, so good. You download the nightly, unpack, ident (the rule to extract the changeset id). Now you back to your source tree and hg update to that revision, and run compare-locales against that. We’d be able to reliably say “works” or “better don’t touch”.

We promised more, and more pieces came together today.

With reliable compare-locales code, one can not only detect missing strings, but also add missing strings to files. Think about a CPP step, nothing permanent, nothing gets landed upstream. But just for the needs of this particular build, you’d have something that has all strings. Not all translated, some padded from en-US. That works. compare-locales is already able to do merges for a while now, storing the changed files into a separate location. Mostly because I consider changing the source to be evil. So what about missing files? Nothing. Good files? Nothing. How does the build pick up files from merges, l10n, and en-US then?

By rewriting make-jars.pl, enter JarMaker.py. Among overall readibility improvements and removing XPFE hacks, JarMaker.py offers to pick up l10n files from a list of top-level source dirs.  It offers another cute feature, by writing both chrome and extension manifests at once. Now, with bug 458014, we don’t have to run the libs phase for installers and langpack separately. (I never got why we do that until I rewrote make-jars.pl, actually.) The rewrite of JarMaker.py was preceeded by rewriting Preprocessor.py, so that all of the jar generation can happen in a single python process.

Starting from today, all of this came together with my installation of buildbot on the l10n server.

This gives us

  • builds on push, i.e., feedback within 5-10 minutes (real stats pending)
  • comparisons of the l10n tip against both
    • the en-US tip (for the upcoming nightly)
    • changeset of the previous (for the existing nightly, with l10n-merge)
  • html-ified output for both of those
  • updates for the dashboard

and last, but not at all least, a

  • working build, even for partial translations.

Find 60 3.1b2pre linux builds on the l10n server.

Thanks to Armen, I used a few of his new makefile targets for download and upload, he did a bunch of work for the sourcestamps-in-application.ini on cvs, too. Thanks to Ted, the poor fellow had to review all my rewrites and Makefile dependencies foo, and did some patches, too. hgpoller stuff not to forget.


  • silme will offer even more reliable merges
  • nightly scheduler for all locales (currently I only build on l10n and en-US l10n-impact changes) (*)
  • mar’s
  • comm-central
  • more Makefile foo to pick up more missing files from en-US in doubt… (*)
  • … or at least document the core set of required files (*)

I won’t take most of those, fwiw. Possibly only the (*) ones.

Sources are in my tooling repository, and there’s an updated version of compare-locales, 0.6, on pypi. No drastic changes here, just some paths fixes, mostly for Windows.

The Conversation {7 comments}

  1. Ville {Tuesday October 21, 2008 @ 10:10 pm}

    Fantastic! A big thank you to all involved!

    I don’t see a mention of any notification / visual alarm that the said l10n files have been amended with en-us strings which was briefly discussed during Armens and John presentations QA sessions. Is it there somewhere or any plans to add something like that?

  2. Axel Hecht {Wednesday October 22, 2008 @ 12:26 am}


    Regarding the branding or other visual feedback, I have that in the back of my head, but it’s not on the top of the list of action items.

  3. Dwayne Bailey {Wednesday October 22, 2008 @ 12:57 am}

    Axel, you know I can’t resist…. :)

    All I can say is finally, we’ve been doing this for the last 5 years, or is it more, with the Translate Toolkit’s moz2po and some build scripts.


    Nice to see we’re moving to a position where we eliminate types of errors. We can finally now move to a concept of priority translations because the build system itself can cope with untranslated text. I hope that that leads to more rapid integration of new languages.

  4. Axel Hecht {Wednesday October 22, 2008 @ 1:21 am}

    Hey Dwayne, I know. And you know that I can’t either ;-), does the translate toolkit work with 3.1 by now?

    On the second item, yes, I am optimistic that this will change the way that folks work on l10n, in particular folks starting from scratch. That said, I don’t see us changing the criteria we have for shipping a locale. That remains to be a full source localization.

  5. Ville {Wednesday October 22, 2008 @ 10:58 am}

    Uh and hah, no need for any branding/visual feedback items needed yet…. At least the Finnish version was pretty busted as I now got to the test it: http://www.flickr.com/photos/vpk/2965000404/

  6. Seth {Thursday October 23, 2008 @ 1:44 am}

    Congratulations on this Axel. This is a big step in the right direction and I know it’s something that you’ve been working on for a long time. I think the part I read from your post that most excites me is

    “and last, but not at all least, a working build, even for partial translations.”

    Seems like this will be one of the things most valuable to localizers trying to get a more real-time and visual status of their work, no?

    More to come, I am sure, but great work so far!

  7. Axel Hecht {Thursday October 23, 2008 @ 2:59 am}

    Thanks, and yes, more to come. Like bug fixes.

    The merges now get cleared between builds, so you don’t end up with unknown language gibberish in your builds. Oops ;-) Thanks to Guillermo for poking. And Ville apparently found the same bug.

Sorry, comments for this entry are closed at this time.