{"id":1686,"date":"2024-05-28T15:00:55","date_gmt":"2024-05-28T15:00:55","guid":{"rendered":"https:\/\/blog.mozilla.org\/l10n\/?p=1686"},"modified":"2026-02-05T11:58:21","modified_gmt":"2026-02-05T11:58:21","slug":"transforming-translations-how-llms-can-help-improve-mozillas-pontoon","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/l10n\/2024\/05\/28\/transforming-translations-how-llms-can-help-improve-mozillas-pontoon\/","title":{"rendered":"Transforming Translations: How LLMs Can Help Improve Mozilla\u2019s Pontoon"},"content":{"rendered":"<div id=\"attachment_1687\" style=\"width: 1034px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image1.png\"><img aria-describedby=\"caption-attachment-1687\" decoding=\"async\" loading=\"lazy\" class=\"wp-image-1687 size-full\" src=\"https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image1.png\" alt=\"A futuristic AI generated city\" width=\"1024\" height=\"1024\" srcset=\"https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image1.png 1024w, https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image1-300x300.png 300w, https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image1-600x600.png 600w, https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image1-150x150.png 150w, https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image1-768x768.png 768w, https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image1-1000x1000.png 1000w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/a><p id=\"caption-attachment-1687\" class=\"wp-caption-text\">Image generated by DALL-E 3<\/p><\/div>\n<p>Imagine a world where language barriers do not exist; a tool so intuitive that it can understand the subtleties of every dialect and the jargon of any industry.<\/p>\n<p>While we&#8217;re not quite there yet, advancements in Large Language Models (LLMs) are bringing us closer to this vision.<\/p>\n<h2>What are LLMs: Beyond the Buzz<\/h2>\n<p>2024 is buzzing with talk about &#8220;AI,&#8221; but what does it actually mean? Artificial Intelligence, especially LLMs, isn&#8217;t just a fad \u2014 it&#8217;s a fundamental shift in how we interface with technology. You&#8217;ve likely interacted with AI without even realizing it \u2014 when Google auto-completes your searches, when Facebook suggests who to tag in a photo, or when Netflix recommends what you should watch next.<\/p>\n<p>LLMs are a breed of AI designed to understand and generate human language by analyzing vast amounts of text. They can compose poetry, draft legal agreements, and yes, translate languages. They&#8217;re not just processing language; they&#8217;re understanding context, tone, and even the subtext of what&#8217;s being written or said.<\/p>\n<h2>The Evolution of Translation: From Machine Translation to LLMs<\/h2>\n<p>Remember the early days of Google Translate? You&#8217;d input a phrase in English and get a somewhat awkward French equivalent. This was typical of statistical machine translation, which relied on vast amounts of bilingual text to make educated guesses. It was magic for its time, but it was just the beginning.<\/p>\n<p>As technology advanced, we saw the rise of neural machine translation, which used AI to better understand context and nuance, resulting in more accurate translations. However, even these neural models have their limitations.<\/p>\n<p>Enter LLMs, which look at the big picture, compare multiple interpretations, and can even consider cultural nuances before suggesting a translation.<\/p>\n<h2>Pontoon: The Heart of Mozilla\u2019s Localization Efforts<\/h2>\n<p>Pontoon isn&#8217;t just any translation tool; it\u2019s the backbone of Mozilla\u2019s localization efforts, where a vibrant community of localizers breathes life into strings of text, adapting Mozilla\u2019s products for global audiences. However, despite integrating various machine translation sources, these tools often struggle with capturing the subtleties essential for accurate translation.<\/p>\n<p>How do we make localizers\u2019 jobs easier? By integrating LLMs to assist not just in translating text but in understanding the spirit of what\u2019s being conveyed. And crucially, this integration doesn&#8217;t replace our experienced localizers who supervise and refine these translations; it supports and enhances their invaluable work.<\/p>\n<h2>Leveraging Research: Making the Case for LLMs<\/h2>\n<p>Our journey began with a question: How can we enhance Pontoon with the latest AI technologies? Diving into research, we explored various LLM applications, from simplifying complex translation tasks to handling under-represented languages with grace.<\/p>\n<p>To summarize the research:<\/p>\n<ul>\n<li aria-level=\"1\"><b>Performance in Translation:<\/b> Studies like &#8220;<a href=\"https:\/\/arxiv.org\/pdf\/2302.14520.pdf\">Large Language Models Are State-of-the-Art Evaluators of Translation Quality<\/a>&#8221; by Tom Kocmi and Christian Federmann demonstrated that LLMs, specifically GPT-3.5 and larger models, exhibit state-of-the-art capabilities in translation quality assessment. These models outperform other automatic metrics in quality estimation without a reference translation, especially at the system level.<\/li>\n<li aria-level=\"1\"><b>Robustness and Versatility: <\/b>The paper &#8220;<a href=\"https:\/\/arxiv.org\/pdf\/2302.09210.pdf\">How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation<\/a>&#8221; by Amr Hendy et al. highlighted the competitive performance of GPT models in translating high-resource languages. It also discussed the limited capabilities for low-resource languages and the benefits of hybrid approaches that combine GPT models with other translation systems.<\/li>\n<li aria-level=\"1\"><b>Innovative Approaches:<\/b> Research on new trends in machine translation, such as \u201c<a href=\"https:\/\/arxiv.org\/pdf\/2305.01181.pdf\">New Trends in Machine Translation using Large Language Models: Case Examples with ChatGPT<\/a>\u201d explored innovative directions like stylized and interactive machine translation. These approaches allow for translations that match specific styles or genres and enable user participation in the translation process, enhancing accuracy and fluency.<\/li>\n<\/ul>\n<p>The findings were clear \u2014 LLMs present a significant opportunity to enhance Pontoon and improve translation quality.<\/p>\n<h2>Why We Chose This Path<\/h2>\n<p>Why go through this transformation? Because language is personal. Take the phrase &#8220;Firefox has your back.&#8221; In English, it conveys reliability and trust. A direct translation might miss this idiomatic expression, interpreting it literally as \u201csomeone has ownership of your back\u201d, which could confuse or mislead users. LLMs can help maintain the intended meaning and nuance, ensuring that every translated phrase feels as though it was originally crafted in the user&#8217;s native language.<\/p>\n<p>We can utilize the in-context learning of LLMs to help with this. This is a technique that informs the model about your data and preferences as it generates its responses via an engineered prompt.<\/p>\n<h2>Experimenting: A Case Study with ChatGPT and GPT-4<\/h2>\n<p>To illustrate the effectiveness of our approach, I conducted a practical experiment with OpenAI\u2019s <a href=\"https:\/\/openai.com\/chatgpt\/\">ChatGPT<\/a>, powered by GPT-4. I asked ChatGPT to translate the string \u201cFirefox has your back\u201d to Bengali. The initial translation roughly translates to \u201cFirefox is behind you\u201d, which doesn\u2019t convey the original meaning of the string.<\/p>\n<div id=\"attachment_1689\" style=\"width: 822px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image3.png\"><img aria-describedby=\"caption-attachment-1689\" decoding=\"async\" loading=\"lazy\" class=\"wp-image-1689 size-full\" src=\"https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image3.png\" alt=\"Screenshot of first interaction with ChatGPT\" width=\"812\" height=\"356\" srcset=\"https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image3.png 812w, https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image3-300x132.png 300w, https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image3-600x263.png 600w, https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image3-768x337.png 768w\" sizes=\"(max-width: 812px) 100vw, 812px\" \/><\/a><p id=\"caption-attachment-1689\" class=\"wp-caption-text\">Asking GPT-4 to translate the string \u201cFirefox has your back\u201d to Bengali.<\/p><\/div>\n<p>Now, it seems our friendly ChatGPT decided to go rogue and translated &#8220;Firefox&#8221; despite being told not to! Additionally, instead of simply providing the translation as requested, it gave a verbose introduction and even threw in an English pronunciation guide. This little mishap underscores a crucial point: the quality of the output heavily depends on how well the input is framed. It appears the AI got a bit too eager and forgot its instructions.<\/p>\n<p>This experiment shows that even advanced models like GPT-4 can stumble if the prompt isn&#8217;t just right. We\u2019ll dive deeper into the art and science of prompt engineering later, exploring how to fine-tune prompts to guide the model towards more accurate and contextually appropriate translations.<\/p>\n<p>Next, I asked ChatGPT to translate the same string to Bengali, this time I specified to keep the original meaning of the string.<\/p>\n<div id=\"attachment_1688\" style=\"width: 866px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image2.png\"><img aria-describedby=\"caption-attachment-1688\" decoding=\"async\" loading=\"lazy\" class=\"wp-image-1688 size-full\" src=\"https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image2.png\" alt=\"Screenshot of second interaction with ChatGPT\" width=\"856\" height=\"394\" srcset=\"https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image2.png 856w, https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image2-300x138.png 300w, https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image2-600x276.png 600w, https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image2-768x353.png 768w\" sizes=\"(max-width: 856px) 100vw, 856px\" \/><\/a><p id=\"caption-attachment-1688\" class=\"wp-caption-text\">Asking GPT-4 to translate the string \u201cFirefox has your back\u201d to Bengali, while maintaining the original meaning of the string.<\/p><\/div>\n<p>Adjusting the prompt, the translation evolved to \u201cFirefox is with you\u201d\u2014a version that better captured the essence of the phrase.<\/p>\n<p>I then used <a href=\"https:\/\/translate.google.ca\/\">Google Translate<\/a> to translate the same string.<\/p>\n<div id=\"attachment_1690\" style=\"width: 713px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image4.png\"><img aria-describedby=\"caption-attachment-1690\" decoding=\"async\" loading=\"lazy\" class=\"wp-image-1690 size-full\" src=\"https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image4.png\" alt=\"Using Google Translate to translate the string \u201cFirefox has your back\u201d to Bengali.\" width=\"703\" height=\"277\" srcset=\"https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image4.png 703w, https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image4-300x118.png 300w, https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image4-600x236.png 600w\" sizes=\"(max-width: 703px) 100vw, 703px\" \/><\/a><p id=\"caption-attachment-1690\" class=\"wp-caption-text\">Using Google Translate to translate the string \u201cFirefox has your back\u201d to Bengali.<\/p><\/div>\n<p>For comparison, Google Translate offered a similar translation to the first attempt by GPT-4, which roughly translates to \u201cFirefox is behind you\u201d. This highlights the typical challenges faced by conventional machine translation tools.<\/p>\n<p>This experiment underscores the potential of stylized machine translation to enhance translation quality, especially for idiomatic expressions or specific styles like formal or informal language.<\/p>\n<h2>The Essential Role of Prompt Engineering in AI Translation<\/h2>\n<p>Building on these insights, we dove deeper into the art of prompt engineering, a critical aspect of working with LLMs. This process involves crafting inputs that precisely guide the AI to generate accurate and context-aware outputs. Effective prompt engineering enhances the accuracy of translations, streamlines the translation process by reducing the need for revisions, and allows for customization to meet specific cultural and stylistic preferences.<\/p>\n<p>Working together with the localization team, we tested a variety of prompts in languages like Italian, Slovenian, Japanese, Chinese, and French. We assessed each translation on its clarity and accuracy, categorizing them as unusable, understandable, or good. After several iterations, we refined our prompts to ensure they consistently delivered high-quality results, preparing them for integration into Pontoon&#8217;s Machinery tab.<\/p>\n<h2>How It Works: Bringing LLMs to Pontoon<\/h2>\n<div id=\"attachment_1691\" style=\"width: 610px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image5.gif\"><img aria-describedby=\"caption-attachment-1691\" decoding=\"async\" loading=\"lazy\" class=\"wp-image-1691 size-full\" src=\"https:\/\/blog.mozilla.org\/l10n\/files\/2024\/05\/image5.gif\" alt=\"LLM feature demonstration\" width=\"600\" height=\"654\" \/><\/a><p id=\"caption-attachment-1691\" class=\"wp-caption-text\">Above is a demonstration of using the \u201cRephrase\u201d option on the string \u201cFirefox has your back\u201d for the Italian locale. The original suggestion from Google\u2019s Machine Translation meant \u201cFirefox covers your shoulders\u201d, while the rephrased version means \u201cFirefox protects you\u201d.<\/p><\/div>\n<p>After working on the prompt engineering and implementation, we&#8217;re excited to announce the integration of LLM-assisted translations into Pontoon. For all locales utilizing Google Translate as a translation source, a new AI-powered option is now available within the \u2018Machinery\u2019 tab \u2014 the reason for limiting the feature to these locales is to gather insights on usage patterns before considering broader integration. Opening this dropdown will reveal three options:<\/p>\n<ul>\n<li aria-level=\"1\"><code>REPHRASE<\/code>: Generate an alternative to this translation.<\/li>\n<\/ul>\n<ul>\n<li aria-level=\"1\"><code>MAKE FORMAL<\/code>: Generate a more formal version of this translation.<\/li>\n<\/ul>\n<ul>\n<li aria-level=\"1\"><code>MAKE INFORMAL<\/code>: Generate a more informal version of this translation.<\/li>\n<\/ul>\n<p>After selecting an option, the revised translation will replace the original suggestion. Once a new translation is generated, another option <code>SHOW ORIGINAL<\/code> will be available in the dropdown menu. Selecting it will revert to the original suggestion.<\/p>\n<h2>The Future of Translation is Here<\/h2>\n<p>As we continue to integrate Large Language Models (LLMs) into Mozilla&#8217;s Pontoon, we&#8217;re not just transforming our translation processes \u2014 we&#8217;re redefining how linguistic barriers are overcome globally. By enhancing translation accuracy, maintaining cultural relevance, and capturing the nuances of language through the use of LLMs, we&#8217;re excited about the possibilities this opens up for users worldwide.<\/p>\n<p>However, it&#8217;s important to emphasize that the role of our dedicated community of localizers remains central to this process. LLMs and machine translation tools are not used without the supervision and expertise of experienced localizers. These tools are designed to support, not replace, the critical work of our localizers who ensure that translations are accurate and culturally appropriate.<\/p>\n<p>We are eager to hear your thoughts. How do you see this impacting your experience with Mozilla&#8217;s products? Do the translations meet your expectations for accuracy? Your feedback is invaluable as we strive to refine and perfect this technology. Please share your thoughts and experiences in the comments below or reach out to us on<a href=\"https:\/\/chat.mozilla.org\/#\/room\/#pontoon:mozilla.org\"> Matrix<\/a>, or<a href=\"https:\/\/github.com\/mozilla\/pontoon\/issues\"> file an issue<\/a>. Together, we can make the web a place without language barriers.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Imagine a world where language barriers do not exist; a tool so intuitive that it can understand the subtleties of every dialect and the jargon of any industry. While we&#8217;re &hellip; <a class=\"go\" href=\"https:\/\/blog.mozilla.org\/l10n\/2024\/05\/28\/transforming-translations-how-llms-can-help-improve-mozillas-pontoon\/\">Read more<\/a><\/p>\n","protected":false},"author":1981,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[467030,137,286406],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/posts\/1686"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/users\/1981"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/comments?post=1686"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/posts\/1686\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/media?parent=1686"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/categories?post=1686"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/l10n\/wp-json\/wp\/v2\/tags?post=1686"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}