The amount of content to translate for Mozilla projects is constantly growing, with overlapping and often demanding deadlines. We want to support our community of volunteers by making their life easier, in particular when it comes to using translations that have already been reviewed and approved before, but also by bootstrapping translation for brand-new content.
As part of this effort, over 3 years ago we started working on the foundation layers to support pretranslation in Pontoon, and we’re almost ready to open this feature up to all supported locales.
Before we dive in, what is pretranslation, and how does it work? If pretranslation is enabled for a combination of locale and project, when a new string is added in Pontoon:
- It will be translated (pretranslated) using a 100% match from translation memory or, should that not be available, using the Google AutoML Translation engine with a custom model.
- The string will be stored in Pontoon with a special “pretranslated” status. The pretranslated status shows up in dashboards, and can be used in filters within the translation interface.
- The string will also be saved in the repository (e.g. GitHub, and eventually ship in the product.
Pretranslation can be enabled for a subset of locales in any project, and the list of locales can differ between projects.
You might be asking why Google AutoML Translation and not another service. We selected this provider based on several criteria, including reliability, quality of results, and range of supported locales. In terms of features, Google AutoML Translation also allows us to fine tune the translation engine by training it on our own existing translation memories, which increases the chances to match the existing style and terminology.
Alpha Testing
To start, we tested pretranslation with only 2 locales — Italian and Slovenian — between December 2022 and March 2023. We picked these locales because we had staff support to review translations, and fixed several bugs in the process.
As part of this phase, we tested pretranslation on 6 different projects (Firefox accounts, Firefox for iOS, Thunderbird, Firefox Monitor, Mozilla VPN Client, Focus for Android), to cover different file formats (GNU gettext, Fluent, XLIFF, Android XML), for a total of 1318 strings.
The results*, especially when accounting for the bugs we fixed over time, were above our initial expectations:
- 65.10% pretranslated strings were approved without changes.
- 94.61% were manually reviewed as “usable”.
For the manual review, we assigned a score between 1 and 3 to the rejected pretranslations:
- 1: Translation is unusable, either because it can’t be understood, or it misleads the user.
- 2: Translation is somehow understandable, but should be improved.
- 3: Translation is good, only needs a minor tweak.
A score of 2 or 3 classifies the rejected pretranslation as “usable”.
The average chF++ score — the algorithm we are using to automatically evaluate translations — was 92.97 (the closer to 100, the better).
The full data from our Alpha testing is available in this spreadsheet.
* The results for the Alpha testing that we published previously, for example in our May l10n report, were calculated incorrectly. The issue was identified and fixed during the Beta phase.
Beta Testing
For the next phase, we planned to expand our testing to Mozilla’s locales with most users (French, German), and add 3 more locales willing to volunteer. We sent a call for locales to opt in at the end of February, and ended up accepting 5 more locales.
Our Beta testing ran between April 1st and June 30th, with:
- 9 locales: Welsh (cy), German (de), Spanish from Argentina (es-AR), French (fr), Hungarian (hu), Indonesian (id), Italian (it), Slovenian (sl), Traditional Chinese (zh-TW).
- 11 projects: AMO Linter, Firefox accounts, Firefox for Android, Firefox for iOS, Firefox Monitor, Firefox Relay (website + add-on), Focus for Android, Mozilla VPN Client, mozilla.org, Thunderbird.
Results
Over 3 months, localizers reviewed 3211 pretranslated strings:
- The average approval rate was 61.10%. The lowest locale had an approval rate of 41.85% (zh-TW), while the highest had 84.29% (de).
- The average review time was about 8 days. Excluding the slowest locale, the average for the remaining 8 locales was 2.5 days, which is an impressive result.
The project with the worst performance was AMO Linter (enabled as a test for Italian), due to the nature of the source content (inconsistencies in terminology and typography, in general developer-focused content difficult to translate). This highlights the impact of the proactive review work that the Localization Team does on projects before content gets exposed for localization in Pontoon, and how that can positively impact pretranslation results.
Mozilla.org also performed below average (average approval at 39.52%), but the data might be skewed by the fact that only 4 locales were enabled, and 3 were the locales with the lowest overall approval rate. On the other hand, projects with short strings (like mobile projects) seemed to perform overall better (approval between 65% and 81%), possibly thanks to a higher number of perfect matches with translation memory.
Out of the 963 rejected strings, 80.89% were marked as “usable” when manually reviewed by localizers. Both Italian (+11.67%) and Slovenian (+11.15%) performed better in Beta than Alpha for this metric, proving the impact of the code improvements made during Beta.
The full data from our Beta testing is available in this spreadsheet, while manual evaluation is available here.
Survey
We also ran a survey to ask all localizers involved how they felt about the feature. These were the main takeaways:
- 91.7% of participants found the feature helpful and would like to use it for more projects.
- 90.9% think we should use pretranslation to bootstrap new projects.
- 63.6% think we should enable pretranslation by default.
Full analysis of the survey results is available here.
Acknowledgements and Next Steps
While this has proven to be a complex feature to implement — and there’s still space for improvement, in particular when it comes to more complex Fluent strings, or evaluating alternative machine translation engines, starting with our own Firefox Translations — it’s important to acknowledge, once again, the fundamental role played by our community. Without their support and their responsiveness during Beta testing, we wouldn’t be able to move forward with the same level of confidence.
What are the next steps? In the coming weeks, we are going to implement an opt-in form in Pontoon, so that locales can request to enable pretranslation directly (see specs here). We’ll make sure to announce it via Discourse and Pontoon notifications as soon as the form is available. We’re also going to add continuous monitoring, showing pretranslation statistics in the Insights dashboards for locales and projects.
Opt-in Guidelines
These are to be considered guidelines more than strict criteria: members of staff will evaluate each request to opt in individually, based on their knowledge of the project and direct experience with the locale.
Criteria for enabling pretranslation for a new locale
- Request needs to come from translators or managers active within the last month (translating or reviewing).
- There is an active manager for the locale (last activity within 2 months).
Criteria for enabling pretranslation for a new project
- Less than 400 missing strings, except for projects or locales where existing pretranslation statistics provide high-confidence.
- Average review time for pretranslations in existing projects is faster than 3 weeks.
Criteria for disabling the feature for a locale or a project
- Approval rate drops below 40%.
- Average review time for pretranslations is slower than 6 weeks.
Note that disabling a project would always involve a conversation with reviewers for the locale.
No comments yet
Post a comment