{"id":1518,"date":"2021-05-01T05:24:56","date_gmt":"2021-05-01T05:24:56","guid":{"rendered":"https:\/\/blog.mozilla.org\/l10n\/?p=1518"},"modified":"2021-05-04T15:00:45","modified_gmt":"2021-05-04T15:00:45","slug":"mozilla-vpn-client-a-localization-tale","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/l10n\/2021\/05\/01\/mozilla-vpn-client-a-localization-tale\/","title":{"rendered":"Mozilla VPN Client: A Localization Tale"},"content":{"rendered":"<p>On April 28th, Mozilla successfully launched its <a href=\"https:\/\/www.mozilla.org\/products\/vpn\/\">VPN Client<\/a> in two new countries: <strong>Germany<\/strong> and <strong>France<\/strong>. While the VPN Client has been available since 2020 in several countries (U.S., U.K., Canada, New Zealand, Singapore, and Malaysia), the user interface was only available in English.<\/p>\n<p>This blog post describes the process and steps needed to make this type of product localizable within the Mozilla ecosystem.<br \/>\n<a href=\"https:\/\/blog.mozilla.org\/l10n\/files\/2021\/05\/italian_screenshot.png\"><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-large wp-image-1519\" src=\"https:\/\/blog.mozilla.org\/l10n\/files\/2021\/05\/italian_screenshot-600x755.png\" alt=\"Screenshot of Mozilla VPN Client with Italian localization\" width=\"600\" height=\"755\" srcset=\"https:\/\/blog.mozilla.org\/l10n\/files\/2021\/05\/italian_screenshot-600x755.png 600w, https:\/\/blog.mozilla.org\/l10n\/files\/2021\/05\/italian_screenshot-252x317.png 252w, https:\/\/blog.mozilla.org\/l10n\/files\/2021\/05\/italian_screenshot-768x967.png 768w, https:\/\/blog.mozilla.org\/l10n\/files\/2021\/05\/italian_screenshot.png 944w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/a><\/p>\n<p><!--more--><\/p>\n<h2>How It Begins<\/h2>\n<p>Back in October 2020, the small team working on this project approached me with a request: we plan to do a complete rewrite of the existing VPN Client with <a href=\"https:\/\/www.qt.io\/\">Qt<\/a>, using one codebase for all platforms, and we want to make it localizable. How can we make it happen?<\/p>\n<p>First of all, let me stress how important it is for a team to <b>reach out as early as possible<\/b>. That allows us to understand existing limitations, explain what we can realistically support, and set clear expectations. It\u2019s never fun to find yourself backed in a corner, late in the process and with deadlines approaching.<\/p>\n<h2>Initial Localization Setup<\/h2>\n<p>This specific project was definitely an interesting challenge, since we didn\u2019t have any prior experience with Qt, and we needed to make sure the project could be supported in <a href=\"https:\/\/pontoon.mozilla.org\">Pontoon<\/a>, our internal Translation Management System (TMS).<\/p>\n<p>The initial research showed that Qt natively uses an XML format (<a href=\"https:\/\/doc.qt.io\/qt-5\/linguist-ts-file-format.html\">TS File<\/a>), but that would have required resources to write a parser and a serializer for Pontoon. Luckily, Qt also supports import and export from a more common standard, <a href=\"https:\/\/doc.qt.io\/qt-5\/linguist-manager.html\">XLIFF<\/a>.<\/p>\n<p>The next step is normally to decide how to structure the content: do we want the TMS to write directly in the main repository, or do we want to use an external repository exclusively for l10n? In this case, we opted for the latter, also considering that the main repository was still private at the time.<\/p>\n<p>Once settled on the format and repository structure, the next step is to do a full review of the existing content:<\/p>\n<ul>\n<li>Check every string for potential localizability issues.<\/li>\n<li>Add comments where the content is ambiguous or there are variables replaced at run-time.<\/li>\n<li>Check consistency issues in the en-US content, in case the content hasn\u2019t been reviewed or created by our very capable Content Team.<\/li>\n<\/ul>\n<p>It\u2019s useful to note that this process heavily depends on the Localization Project Manager assigned to a project, because there are different skill sets in the team. For example, I have a very hands-on approach, often writing patches directly to fix small issues like missing comments (that normally helps reducing the time needed for fixes).<\/p>\n<p>In my case, this is the ideal approach:<\/p>\n<ul>\n<li>After review, set up the project in Pontoon as a private project (only accessible to admins).<\/li>\n<li>Actually translate the project into Italian. That allows me to verify that everything is correctly set up in Pontoon and, more importantly, it allows me to identify issues that I might have missed in the initial review. It\u2019s amazing how differently your brain works when you\u2019re just looking at content, and when you\u2019re actually trying to translate it.<\/li>\n<li>Test a localized build of the product. In this way I can verify that we are able to use the output of our TMS, that the build system works as expected, and that there are no errors (hard-coded content, strings reused in different contexts, etc.).<\/li>\n<\/ul>\n<p>This whole process typically requires at least a couple of weeks, depending on how many other projects are active at the same time.<\/p>\n<h2>Scale and Automate<\/h2>\n<p>I\u2019m a huge fan of automation when it comes to getting rid of repetitive tasks, and I\u2019ve come to learn a lot about GitHub Actions working on this project. Luckily, that knowledge helped in several other projects later on.<\/p>\n<p>The first thing I noticed is that I was often commenting on two issues on the source (en-US) strings: typographic issues (straight quotes, 3 dots instead of ellipsis), lack of comments when a string has variables. So I wrote a very <a href=\"https:\/\/github.com\/mozilla-mobile\/mozilla-vpn-client\/blob\/main\/.github\/l10n\/check_l10n_issues.py\">basic linter<\/a> that runs in automation every time a developer adds new strings in a pull request.<\/p>\n<p>The bulk of the automation lives in the l10n repository:<\/p>\n<ul>\n<li>There\u2019s <a href=\"https:\/\/github.com\/mozilla-l10n\/mozilla-vpn-client-l10n\/blob\/main\/.github\/workflows\/update.yaml\">automation<\/a>, running daily, that extracts strings from the code repository, and creates a PR exposing them to all locales.<\/li>\n<li>There\u2019s a <a href=\"https:\/\/github.com\/mozilla-l10n\/mozilla-vpn-client-l10n\/blob\/main\/.github\/scripts\/check_locales.py\">basic linter<\/a> that checks for issues in the localized content, in particular missing variables. That happens more often than it should, mostly because the placeholder format is different from what localizers are used to, and there might be Translation Memory matches \u2014 strings already translated in the past in other products \u2014 coming from different file formats.<\/li>\n<\/ul>\n<p><a href=\"https:\/\/blog.mozilla.org\/l10n\/files\/2021\/05\/vpn.png\"><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-large wp-image-1520\" src=\"https:\/\/blog.mozilla.org\/l10n\/files\/2021\/05\/vpn-600x979.png\" alt=\"VPN L10n Workflow Diagram\" width=\"600\" height=\"979\" srcset=\"https:\/\/blog.mozilla.org\/l10n\/files\/2021\/05\/vpn-600x979.png 600w, https:\/\/blog.mozilla.org\/l10n\/files\/2021\/05\/vpn-252x411.png 252w, https:\/\/blog.mozilla.org\/l10n\/files\/2021\/05\/vpn.png 760w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/a>The update automation was particularly interesting. <a href=\"https:\/\/github.com\/mozilla-l10n\/mozilla-vpn-client-l10n\/blob\/main\/.github\/scripts\/extract_source_strings.py\">Extracting<\/a> new en-US strings is relatively easy, thanks to Qt command line tools, although there is some work needed to clean up the resulting XLIFF (for example, moving localization comments from <code>extracomment<\/code> to <code>note<\/code>).<\/p>\n<p>In the process of adding new locales, we quickly <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=1676336\">realized<\/a> that updating only the reference file (en-US) was not sufficient, because Pontoon expects each localized XLIFF to have all source messages, even if untranslated.<\/p>\n<p>Historically that was the case for other bilingual file formats  \u2014 files that contain both source and translation \u2014 like .po (GetText) and .lang files, but it is not necessarily true for XLIFF files. In particular, both those formats come with their own set of tools to merge new strings from a template into other locales, but that\u2019s not available for XLIFF, which is an exchange format used across completely different tools.<\/p>\n<p>At this point, i needed automation to solve two separate issues:<\/p>\n<ul>\n<li>Add new strings to all localized files when updating en-US.<\/li>\n<li>Catch unexpected <a href=\"https:\/\/mozilla-l10n.github.io\/documentation\/localization\/making_string_changes.html\">string changes<\/a>. If a string changes without a new ID, it doesn\u2019t trigger any action in Pontoon (existing translations are kept, localizers won\u2019t be aware of the change). So we need to make sure those are correctly managed.<\/li>\n<\/ul>\n<p>This is how a string looks like in the source XLIFF file:<\/p>\n<pre>\r\n&lt;file original=\"..\/src\/ui\/components\/VPNAboutUs.qml\" datatype=\"plaintext\"&gt;\r\n  &lt;body&gt;\r\n    &lt;trans-unit id=\"vpn.aboutUs.tos\"&gt;\r\n      &lt;source&gt;Terms of Service&lt;\/source&gt;\r\n    &lt;\/trans-unit&gt;\r\n  &lt;\/body&gt;\r\n&lt;\/file&gt;\r\n<\/pre>\n<p>These are the main steps in the <a href=\"https:\/\/github.com\/mozilla-l10n\/mozilla-vpn-client-l10n\/blob\/main\/.github\/scripts\/update_other_locales.py\">update script<\/a>:<\/p>\n<ul>\n<li>It takes the en-US XLIFF file, and uses it as a template.<\/li>\n<li>It reads each localized file, saving existing translations. These are stored in a dictionary, where the key is generated using the original attribute of the <code>file<\/code> element, the string ID from the <code>trans-unit<\/code>, and a hash of the actual source string.<\/li>\n<li>Translations are then injected in the en-US template and saved, overwriting the existing localized file.<\/li>\n<\/ul>\n<p>Using the en-US file as template ensures that the file includes all the strings. Using the hash of the source text as part of the ID will remove translations if the source string changed (there won\u2019t be a translation matching the ID generated while walking through the en-US file).<\/p>\n<h2>Testing<\/h2>\n<p>How do you test a project that is not publicly available, and requires a paid subscription on top of that? Luckily, the team came up with the brilliant idea of creating a <a href=\"https:\/\/mozilla-mobile.github.io\/mozilla-vpn-client\/\">WASM online application<\/a> to allow our volunteers to test their work, including parts of the UI or dialogs that wouldn\u2019t be normally exposed in the main user interface.<\/p>\n<p>Localized strings are automatically imported in the build process (the l10n repository is configured as a submodule in the code repository), and screenshots of the app are also generated as part of the automation.<\/p>\n<h2>Conclusions<\/h2>\n<p>This was a very interesting project to work on, and I consider it to be a success case, especially when it comes to cooperation between different teams. A huge thanks to <a href=\"https:\/\/github.com\/bakulf\/\">Andrea<\/a>, <a href=\"https:\/\/github.com\/lesleyjanenorton\">Lesley<\/a>, <a href=\"https:\/\/github.com\/strseb\">Sebastian<\/a> for being always supportive and helpful in this long process, and constantly caring about localization.<\/p>\n<p>Thanks to the amazing work of <b>our community of localizers<\/b>, we were able to exceed the minimum requirements (support French and German): on launch day, Mozilla VPN Client was available in <b>25 languages<\/b>.<\/p>\n<p>Keep in mind that this was only one piece of the puzzle in terms of supporting localization of this product: there is web content localized as part of mozilla.org, parts of the authentication flow managed in a different project, payment support in Firefox Accounts, legal documents and user documentation localized by vendors, and SUMO pages.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>On April 28th, Mozilla successfully launched its VPN Client in two new countries: Germany and France. While the VPN Client has been available since 2020 in several countries (U.S., U.K., &hellip; <a class=\"go\" href=\"https:\/\/blog.mozilla.org\/l10n\/2021\/05\/01\/mozilla-vpn-client-a-localization-tale\/\">Read more<\/a><\/p>\n","protected":false},"author":663,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12679,199,137,136095],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/posts\/1518"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/users\/663"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/comments?post=1518"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/posts\/1518\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/media?parent=1518"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/categories?post=1518"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/tags?post=1518"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}