{"id":1653,"date":"2024-02-07T10:55:23","date_gmt":"2024-02-07T10:55:23","guid":{"rendered":"https:\/\/blog.mozilla.org\/l10n\/?p=1653"},"modified":"2024-02-23T15:33:04","modified_gmt":"2024-02-23T15:33:04","slug":"a-deep-dive-into-the-evolution-of-pretranslation-in-pontoon","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/l10n\/2024\/02\/07\/a-deep-dive-into-the-evolution-of-pretranslation-in-pontoon\/","title":{"rendered":"A Deep Dive Into the Evolution of Pretranslation in Pontoon"},"content":{"rendered":"

Quite often, an imperfect translation is better than no translation. So why even publish untranslated content when high-quality machine translation systems are fast and affordable? Why not immediately machine-translate content and progressively ship enhancements as they are submitted by human translators?<\/p>\n

At Mozilla, we call this process pretranslation<\/i>. We began implementing it in Pontoon before COVID-19 hit, thanks to Vishal<\/a> who landed the first patches. Then we caught some headwinds and didn\u2019t make much progress until 2022 after receiving a significant development boost and finally launched it for the general audience in September 2023.<\/p>\n

So far, 20 of our localization teams (locales) have opted to use pretranslation across 15 different localization projects. Over 20,000 pretranslations have been submitted and none of the teams have opted out of using it. These efforts have resulted in a higher translation completion rate, which was one of our main goals.<\/p>\n

In this article, we\u2019ll take a look at how we developed pretranslation in Pontoon. Let\u2019s start by exploring how it actually works.<\/p>\n

How does pretranslation work?<\/h2>\n

Pretranslation is enabled upon a team\u2019s request (it\u2019s off by default). When a new string is added to a project, it gets automatically pretranslated using a 100% match from translation memory (TM), which also includes translations of glossary entries. If a perfect match doesn\u2019t exist, a locale-specific machine translation (MT) engine is used, trained on the locale\u2019s translation memory.<\/p>\n

\"Pretranslation<\/a>

Pretranslation <\/i>opt-in form<\/i><\/a>.<\/i><\/p><\/div>\n

After pretranslations are retrieved and saved in Pontoon, they get synced to our primary localization storage (usually a GitHub repository) and hence immediately made available for shipping. Unless they fail our quality checks. In that case, they don\u2019t propagate to repositories until errors or warnings are fixed during the review process.<\/p>\n

Until reviewed, pretranslations are visually distinguishable from user-submitted suggestions and translations. This makes post-editing much easier and more efficient. Another key factor that influences pretranslation review time is, of course, the quality of pretranslations. So let\u2019s see how we picked our machine translation provider.<\/p>\n

Choosing a machine translation engine<\/h2>\n

We selected the machine translation provider based on two primary factors: quality of translations and the number of supported locales. To make translations match the required terminology and style as much as possible, we were also looking for the ability to fine-tune the MT engine by training it on our translation data.<\/p>\n

In March 2022, we compared Bergamot, Google\u2019s Cloud Translation API (generic), and Google\u2019s AutoML Translation (with custom models). Using these services we translated a collection of 1,000 strings into 5 locales (it, de, es-ES, ru, pt-BR), and used automated scores (BLEU<\/a>, chrF++<\/a>) as well as manual evaluation to compare them with the actual translations.<\/p>\n

\"Performance<\/a>

Performance of tested MT engines for Italian (it).<\/i><\/p><\/div>\n

Google\u2019s AutoML Translation outperformed the other two candidates in virtually all tested scenarios and metrics, so it became the clear choice. It supports over 60 locales. Google\u2019s Generic Translation API supports twice as many, but we currently don\u2019t plan to use it for pretranslation in locales not supported by Google\u2019s AutoML Translation.<\/p>\n

Making machine translation actually work<\/h2>\n

Currently, around 50% of pretranslations generated by Google\u2019s AutoML Translation get approved without any changes. For some locales, the rate is around 70%. Keep in mind however that machine translation is only used when a perfect translation memory match isn\u2019t available. For pretranslations coming from translation memory, the approval rate is 90%.<\/p>\n

\"Comparison<\/a>

Comparison of <\/i>pretranslation approval rate<\/i><\/a> between teams.<\/i><\/p><\/div>\n

To reach that approval rate, we had to make a series of adjustments to the way we use machine translation.<\/p>\n

For example, we convert multiline messages to single-line messages before machine-translating them. Otherwise, each line is treated as a separate message and the resulting translation is of poor quality.<\/p>\n

Multiline message:<\/i>
\n<\/code><\/p>\n

Make this password unique and different from any others you use.\r\nA good strategy to follow is to combine two or more unrelated\r\nwords to create an entire pass phrase, and include numbers and symbols.<\/pre>\n
Multiline message converted to a single-line message:<\/i>
\n<\/code><\/p>\n
Make this password unique and different from any others you use. A good strategy to follow is to combine two or more unrelated words to create an entire pass phrase, and include numbers and symbols.<\/p>\n
Let\u2019s take a closer look at two of the more time-consuming changes.<\/p>\n
The first one is specific to our machine translation provider (Google\u2019s AutoML Translation). During initial testing, we noticed it would often take a long time for the MT engine to return results, up to a minute. Sometimes it even timed out! Such a long response time not only slows down pretranslation, it also makes machine translation suggestions in the translation editor less useful – by the time they appear, the localizer has already moved to translate the next string.<\/p>\n
After further testing, we began to suspect that our custom engine shuts down after a period of inactivity, thus requiring a cold start for the next request. We contacted support and our assumption was confirmed. To overcome the problem, we were advised to send a dummy query to the service every 60 seconds just to keep the system alive.<\/p>\n
\"Giphy:<\/a>
Image source: Giphy<\/a>.<\/p><\/div>\n
Of course, it\u2019s reasonable to shut down inactive services to free up resources, but the way to keep them alive isn\u2019t. We have to make (paid) requests to each locale\u2019s machine translation engines every minute just to make sure they work when we need them. And sometimes even that doesn\u2019t help – we still see about a dozen ServiceUnavailable<\/i> errors every day. It would be so much easier if we could just customize the default inactivity period or pay extra for an always-on service.<\/p>\n
The other issue we had to address is quite common in machine translation systems: they are not particularly good at preserving placeholders<\/a>. In particular, extra space often gets added to variables or markup elements, resulting in broken translations.<\/p>\n
Message with variables:<\/i><\/p>\n
{ $partialSize } of { $totalSize }<\/pre>\n
Message with variables machine-translated to Slovenian (adding space after $ breaks the variable):<\/i><\/p>\n
{$ partialSize} od {$ totalSize}<\/pre>\n
We tried to mitigate this issue by wrapping placeholders in <span translate=”no”>…<\/span>, which tells Google\u2019s AutoML Translation to not translate the wrapped text<\/a>. This approach requires the source text to be submitted as HTML (rather than plain text), which triggers a whole new set of issues \u2014 from adding spaces in other places to escaping quotes \u2014 and we couldn\u2019t circumvent those either. So this was a dead-end.<\/p>\n
The solution was to store every placeholder in the Glossary<\/a> with the same value for both source string and translation. That approach worked much better and we still use it today. It\u2019s not perfect, though, so we only use it to pretranslate strings for which the default (non-glossary) machine translation output fails our placeholder quality checks.<\/p>\n
Making pretranslation work with Fluent messages<\/h2>\n
On top of the machine translation service improvements we also had to account for the complexity of Fluent messages, which are used by most of the projects we localize at Mozilla. Fluent<\/a> is capable of expressing virtually any imaginable message, which means it is the localization system you want to use if you want your software translations to sound natural.<\/p>\n
As a consequence, Fluent message format comes with a syntax that allows for expressing such complex messages. And since machine translation systems (as seen above) already have trouble with simple variables and markup elements, their struggles multiply with messages like this:
\n<\/code><\/p>\n
shared-photos =\r\n { $photoCount ->\r\n    [one]\r\n      { $userGender ->\r\n        [male] { $userName } added a new photo to his stream.\r\n        [female] { $userName } added a new photo to her stream.\r\n       *[other] { $userName } added a new photo to their stream.\r\n      }\r\n   *[other]\r\n      { $userGender ->\r\n        [male] { $userName } added { $photoCount } new photos to his stream.\r\n        [female] { $userName } added { $photoCount } new photos to her stream.\r\n       *[other] { $userName } added { $photoCount } new photos to their stream.\r\n      }\r\n  }<\/pre>\n
That means Fluent messages need to be pre-processed before they are sent to the pretranslation systems. Only relevant parts of the message need to be pretranslated, while syntax elements need to remain untouched. In the example above, we extract the following message parts, pretranslate them, and replace them with pretranslations in the original message:<\/p>\n