{"id":81570,"date":"2025-08-28T10:51:24","date_gmt":"2025-08-28T17:51:24","guid":{"rendered":"https:\/\/blog.mozilla.org\/?p=81570"},"modified":"2025-10-01T09:19:01","modified_gmt":"2025-10-01T16:19:01","slug":"speeding-up-firefox-local-ai-runtime","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/en\/firefox\/firefox-ai\/speeding-up-firefox-local-ai-runtime\/","title":{"rendered":"Speeding up Firefox Local AI Runtime"},"content":{"rendered":"\n

Last year we rolled out the Firefox AI Runtime<\/a><\/strong>, the engine that quietly powers features such as PDF.js generated alt text<\/a> and, more recently, our smart tab grouping<\/em><\/a>. The system worked, but not quite at the speed we wanted.<\/p>\n\n\n\n

This post explains how we accelerated inference by replacing the default onnxruntime\u2011web<\/a><\/strong> that powers Transformers.js<\/a> with its native C++ counterpart that now lives inside Firefox.<\/p>\n\n\n\n

Where we started<\/h2>\n\n\n\n

Transformers.js is the JavaScript counterpart to Hugging Face\u2019s Python library. Under the hood it relies on onnxruntime\u2011web<\/strong>, a WebAssembly (WASM) build of ONNX Runtime.<\/p>\n\n\n\n

A typical inference cycle:<\/p>\n\n\n\n

    \n
  1. Pre\u2011processing<\/strong> in JavaScript (tokenization, tensor shaping)<\/li>\n\n\n\n
  2. Model execution<\/strong> in WASM<\/li>\n\n\n\n
  3. Post\u2011processing<\/strong> back in JavaScript<\/li>\n<\/ol>\n\n\n\n

    Even with warm caches, that dance crosses multiple layers. The real hotspot is the matrix multiplications, implemented with generic SIMD when running on CPU.<\/p>\n\n\n\n

    Why plain WASM wasn\u2019t enough<\/h2>\n\n\n\n

    WASM SIMD is great, but it can\u2019t beat hardware\u2011specific instructions such as NEON on Apple Silicon or AVX\u2011512 on modern Intel chips.<\/p>\n\n\n\n

    Firefox Translations<\/a> (uses Bergamot<\/em>) already proves that diving to native code speeds things up: it uses WASM built\u2011ins<\/strong> which are small hooks that let WASM call into C++ compiled with those intrinsics. The project, nicknamed gemmology<\/a><\/em>, works brilliantly.<\/p>\n\n\n\n

    We tried porting that trick to ONNX, but the huge number of operators made a one\u2011by\u2011one rewrite unmaintainable. And each cold start still paid the JS\/WASM warm\u2011up tax.<\/p>\n\n\n\n

    Switching to ONNX C++<\/h2>\n\n\n\n

    Transformers.js talks to ONNX Runtime through a tiny<\/em> surface. It creates a session, pushes a Tensor, and pulls a result. It makes it simple to swap the backend without touching feature code.<\/p>\n\n\n\n

    Our steps to achieve this were:<\/p>\n\n\n\n

      \n
    1. Vendor ONNX Runtime C++<\/strong> directly into the Firefox tree.<\/li>\n\n\n\n
    2. Expose<\/strong> it to JavaScript via a thin WebIDL layer.<\/li>\n\n\n\n
    3. Wire<\/strong> Transformers.js to the new backend.<\/li>\n<\/ol>\n\n\n\n

      From the perspective of a feature like PDF alt\u2011text, nothing changed, it still calls await pipeline(\u2026). Underneath, tensors now go straight to native code.<\/p>\n\n\n\n

      Integration of ONNX Runtime to the build system<\/h2>\n\n\n\n

      Upstream ONNX runtime does not support all of our build configuration, and it\u2019s a large amount of code. As a consequence we chose not to add it in-tree. Instead, a configuration flag can be used to provide a compiled version of the ONNX runtime. It is eventually automatically downloaded from Taskcluster<\/a> (where we build it for a selection of supported configuration) or provided by downstream developers. This provides flexibility while not slowing down our usual build and requiring low maintenance.<\/p>\n\n\n\n

      Building ONNX on Taskcluster required some configuration changes and upstream patches. The goal was to find a balance between speed and binary size, while being compatible with native code requirements from the Firefox repo. <\/p>\n\n\n\n

      Most notably:<\/p>\n\n\n\n