Axel Hecht Mozilla in Your Language

February 5, 2010

L20n presentation at FOSDEM 2010

Filed under: L10n,Mozilla — Tags: , , — Axel Hecht @ 4:26 am

I’ll be presenting at FOSDEM on l20n, the infrastructure that we’re hoping to move our localization efforts to. The talk will be in the Mozilla Developers room on Sunday, 13:15. The FOSDEM program might still give a Sunday morning time, that changed.

I’ll focus on what tools can do to help localizers to use the power that l20n brings, without making things totally obscure. I’ll start with a quick recap for those that are new to it, and then discuss the challenges that l20n brings, and how tools can help. I’ll also present first thoughts on how to communicate data describing languages between tools, using html5 and microdata.

See you in Brussels, not just for that talk, of course.

PS: I’ll be giving a lightning talk on the new l10n site, too.

November 27, 2009

hg annotate and “did we ship that?”

Filed under: Mozilla — Tags: , — Axel Hecht @ 9:56 am

I came across the question on whether a particular piece of code was in a particular release, and if so, which. Say, you’re going through old bugs like me and you want to know if something’s fixed b2 or fixed b4. So you head over to hg annotate and then…

I wrote a jetpack feature to the rescue. It’s as rude as it’s icon, but it’s getting the tags and the revisions from hg.m.o, and replaces the not-so-useful author@rev integers with the latest release tag. Currently I hacked this up for Firefox and Thunderbird, and it relies on FIREFOX_version_RELEASE being a proper version. It’s doing two background http requests, so it takes a while, no UI feedback on that. How’d a jetpack do that right?

HTH, and I wish jetpacks were even more simple. Docs, please. And it took me a while to figure out that using the included “Develop” tab isn’t as fancy, editing outside and refresh felt quicker.

Anyway, if you have jetpack installed, head over to the install page of the annotate jetpack feature.

November 12, 2009

Crowdsourcing … exactly what?

Filed under: L10n,Mozilla — Tags: , , — Axel Hecht @ 9:45 am

I’ve just run across an interesting suggestion for translating “Smiley” into English. Screenshot of it would be

Crowd sourcing exactly what?

whereas the original (triple-licensed) translation suggestion is on l10n.mozilla.org/narro.

Another interesting aspect of crowd sourcing, box-of-chocolates style. You never know what you get.

November 6, 2009

PS: l10n-merge

Filed under: L10n,Mozilla — Tags: , , — Axel Hecht @ 8:02 am

Armen just blogged about this, and as it’s constantly mentioned around l10n, I wanted to add a bit more detail to l10n-merge.

l10n-merge is originally an idea by our Japanese localizer dynamis. The current implementation used in the builds is by me, integrated as an option to compare-locales. There are spin-offs of that algorithm in the silme library, too.

l10n-merge attempts to solve one reason for “yellow screens of death”, i.e., XML parsing errors triggered by incomplete localizations. This is really crucial as localizations don’t just pop up by swinging magic wands, they’re incremental work, and a huge chunk of that. So in order to test your work, you need to see the strings you have in, say, Firefox, without having the other 4000 strings done yet. Other l10n-infrastructures handle this by falling back to the original language at runtime (gettext), but doing that at runtime of course has perf impact, and size. l10n-merge does the same thing at compile (repackaging) time.

Design goals for l10n-merge were:

  • not mess with any source repositories
  • not do any file-io that’s not really needed

Thus, in order to not mess with the source repos, l10n-merge doesn’t modify the sources inline, but creates copies of the files it touches in a separate dir. Commonly, we’re using ‘merged‘ in the build dir. Now, creating a full copy of everything would be tons of file io, so l10n-merge only creates copies for those files which actually need to get entities added to existing localized content. This plays together with code in JarMaker.py which is able to pick up locale chrome content from several source dirs.

A Firefox localization contains some 450 files, and say for the current 9 B1-to-B2 missing strings in two files, it would copy over those two files from l10n, and add the missing entities to the end. Then JarMaker is called with the right options, and for those two files, will pick them up from merged, the rest of the localization is gotten from l10n. For missing files, it actually looks into the en-US sources, too, so we don’t have to do anything for those. To give an example, for chrome/browser/foo in the browser ‘module’, it searches:

  1. .../merged/browser/chrome/foo
  2. l10n/ab-CD/browser/chrome/foo
  3. mozilla/browser/locales/en-US/chrome/foo

Now it’s time to list some pitfalls that come with l10n-merge:

  • If you’re passing the wrong dir for mergedir, nothing breaks. All build logic breakage would come from missing files, and due to the fallback to en-US, there are no missing files.
  • l10n-merge, as compare-locales, doesn’t cover XML parsing errors inside entity values yet. Bug 504339 is filed, there are some tricky questions on reporting, as well as having to write an XML parser from scratch.
  • l10n-merge only appends entities, but that’s fine 95% of the time. Only counter-examples are DTDs including other DTDs.
  • People using l10n-merge need to manually maintain the merge dir. Pruning it via compare-locales is risky business if you specify the wrong path by accident, so I consider this a feature. But if you’re seeing Spanish in a French build, clobber the mergedir and build again :-)

September 22, 2009

Axel loves the web

Filed under: Mozilla — Tags: — Axel Hecht @ 10:49 am

Good for the planet earth, I didn’t print poster number 5. I’ve put it up on people.mozilla.org, hopefully that’s serving it with a mime type that’s full of love.

Bad for planet mozilla, I did take a picture and here I’m blogging about it.

Axel loves the web

Yes, I love the web so much, you can hardly see me. But you do see real full screen on a mac, German namoroka nightly. And you the same one web on my N810 (not so current 1.9.2 nightly). And the infamous “Don’t hurt the web” shirt with that, of course.

September 9, 2009

String freezes on 1.9.2

Filed under: L10n,Mozilla — Tags: , — Axel Hecht @ 2:59 pm

As we’re struggling with our string freezes for 1.9.2, I figured I’d put something up on planet for everyone to notice:

  • String freezes are one only, and done with. No “I’ll get that in the next milestone”. First string freeze is the last.
  • String freeze for mobile-browser and toolkit was last Friday. For mobile-browser seriously this Friday, Sept 11th. (*).
  • String freeze for Firefox 3.6 is Sept 14th.

August 20, 2009

Hacking DTDs into Bespin

Filed under: Mozilla — Tags: , — Axel Hecht @ 8:40 am

Just a quick side note, I’m hacking on a DTD mode for Bespin. The code is pretty ughly still, but the output is getting OK.

screenshot of editing a DTD in Bespin

It already supports marking up errors, to a similar extent that they’re found in compare-locales these days. Better parsing for the actual values is a bigger task, in particular one that should improve Bespin’s XML mode at the same time. ‘Cause that didn’t impress me yet.

Code will come in bug 510579.

July 12, 2009

oh my eyes, but good still

Filed under: Mozilla — Tags: , , — Axel Hecht @ 2:19 pm

Builds for a changeI’ve uploaded a snapshot of a build in progress on my test display which gives a bit better insight on what’s possible to show about a build based on the information that buildbot has, or a build database had. The important pieces here compared to tinderbox are:

  • Builds in progress are associated with the check-in that triggered the build.
  • Builds in progress show individual build steps.
  • Finished builds are displayed in compact form. In this case, all builds end with warnings, and thus come back in a shade of orange.
  • Builds not yet started that are already requested are displayed on top.

I didn’t go into the detail of mentioning which builders are having pending builds. This is mostly me waiting for django 1.1 and aggregation support, but in the end it’s as simple as a GROUP BY. Nor did I try to make that display visually pretty, hell no.

In the context of our regular builds, it’s worthwhile mentioning what unit test and talos runs would look like, i.e., “builds” that are scheduled after the binary builds are done. Those wouldn’t show up until they’re actually scheduled, which is fair enough, as that is what’s actually happening. If a windows build fails, there won’t be unit tests nor talos builds run. You wouldn’t end up in a situation where you think you’re done and you aren’t, though. (Talos not working this way aside, that needs thought due to different masters etc.) The actual builders don’t finish until they triggered the spin-off runs, so you either see the binary build as still running, or you see the triggered runs as pending. Here, as soon as you don’t have anything running or pending and your boxens are green, you’re off the hook.

I also added microsummaries and RSS feeds for this view, so you can use the web to learn about your the fire you lit up.

June 12, 2009

150/2=73

Filed under: L10n,Mozilla — Tags: , — Axel Hecht @ 9:51 am

Our brave build folks have cut the tags on Firefox 3.5 RC 1, and I figured I give a little feedback on that from the l10n side.

As RC 1 was based on new strings, we required each localization to sign-off on the status of their localization to be ready for release. We’re still doing this by opening what we call a “opt-in thread”, a message sent to .l10n after the last l10n-impact landing to which localizers reply with a link to the revision of their localization that is good to go. Part of that communication is the message when code-freeze is planned to be, and the message that plans don’t always work out. So we’re keeping the the opt-in thread open actually up to the point where we really kick off the builds.

The output of that process are two files which control our release automation process, shipped-locales and l10n-changesets. For the curious, we’re tracking which locales ship on which platforms in the first, and it’s part of the code repo, and which locales ship which hg revision in the second, which is in the buildbot-configs repo.

The whole process lead to 150 different opt-in sourcestamps which came in by either public replies in the newsgroup, or as private mails in my inbox (or both). I pick those up, and click on some buttons on a version of the l10n dashboard running on my home machine, review the changes to previous sign-offs (yes, I do have a web interface that does comparison between two revisions in hg), and accept or reject the sign-off. If I reject, I follow up in .l10n with why I did that. That adds up to 159 posts in that thread, by 72 authors. Dependent on how imminent the release is, or seems to be, I “back up” my local data by attaching files to the tracking bug. This led to one version of shipped-locales, and a whopping 16 versions of l10n-changesets. Or, in short …

<bhearsum|afk> Pike: when should i expect an updated l10n-changestets?
<Pike> bhearsum: …
<Pike> …
<Pike> …
<Pike> now
<bhearsum> heh
<bhearsum> cool!

What’s really cool here is that we’re actually at a point where we pick up improvements to our localization up to the last minute, with tools that make us feel comfortable about that, and with a release environment that is able to digest all that noise and produce builds for 73 localizations in a matter of a few hours.

73

June 10, 2009

Running into builds, just testing

Filed under: L10n,Mozilla — Tags: , , , — Axel Hecht @ 10:29 am

I’ve blogged previously on how to set up a staging environment to test the l10n build system, but I didn’t go into any detail on how to actually do builds in that set up. That shall be fixed.

I’m picking you up at the point where

twistd get-pushes -n -t 1 -s l10n_site.settings

is running stable. You probably want to tail -f twistd.log to follow what it’s doing. This piece is going to watch the hg repositories in stage/repos and feed all the pushes to those into the db. Next is to make sure that you actually get builds.

The first thing you need to do is to configure the l10n-master to access the same database as the django-site. Just make sure that DATABASE_* settings in l10n-master/settings.py and django-site/l10n_site/settings.py match. The other setting to sync is REPOSITORY_BASE, which needs to match in both settings.pys. I suggest setting that to an empy dir next to the l10n-master. I wouldn’t use the stage/repos dir, mostly because I didn’t test that. Now you set up the master for buildbot, by running

buildbot create-master .

inside the l10n-master clone. Next thing is to create a slave dir, which is well put next to the l10n-master. Despite what buildbot usually likes, this slave should be on the same machine that the rest is running on.

mkdir slave
buildbot create-slave slave localhost:9021 cs pwd

So much for the general setup. With the twistd daemon running to get the pushes, you can now switch on the master and the slave, by doing a buildbot start . in both dirs. I suggest that you keep a tail -f twistd.log running on the master. If you decide to set things up to track the upstream repositories, I start the master, then the slave, and if both are up fine, I start the twistd daemon for get-pushes.

Now let’s trigger some builds:

Open stage/workdir/mozilla/browser/locales/en-US/file.properties in your favorite editor, and do a modification. I suggest to just do a whitespace edit, or to change the value of the existing entity, as that is going to keep your localizations green. Check the change in to the working clone, and then push. The get-pushes output should show that it got a new push, and then on the next cycle, feed the push into the database. You’ll notice by the output of a hg pull showing up in the log. On the next poll on the l10n-master, builds for all 4 locales should trigger. You should see an update of four builds on the waterfall, and 4 locales on the test tree on the local dashboard.

« Newer Posts — Older Posts »

Powered by WordPress