{"id":459,"date":"2025-12-09T15:40:28","date_gmt":"2025-12-09T15:40:28","guid":{"rendered":"https:\/\/blog.mozilla.org\/data\/?p=459"},"modified":"2025-12-09T15:40:28","modified_gmt":"2025-12-09T15:40:28","slug":"incident-report-a-compiler-bug-and-json","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/data\/2025\/12\/09\/incident-report-a-compiler-bug-and-json\/","title":{"rendered":"Incident Report: A compiler bug and JSON"},"content":{"rendered":"

It all started rather inconspicuous: The Data Engineering team filed a bug report<\/a> about a sudden increase in schema errors at ingestion of telemetry data from Firefox for Android. At that point in time about 0.9% of all incoming pings were not passing our schema validation checks.<\/p>\n

The data we were seeing was surprising. Our ingestion endpoint received valid JSON that contained snippets like this:<\/p>\n

\n
{<\/span><\/span>\r\n    \"metrics\"<\/span>:<\/span> {<\/span><\/span>\r\n        \"schema: counter\"<\/span>:<\/span> {<\/span><\/span>\r\n            \"glean.validation.pings_submitted\"<\/span>:<\/span> {<\/span><\/span>\r\n                \"events\"<\/span>:<\/span> 1<\/span><\/span>\r\n            }<\/span><\/span>\r\n        },<\/span><\/span>\r\n        ...<\/span><\/span>\r\n    },<\/span><\/span>\r\n    ...<\/span><\/span>\r\n}<\/span><\/span><\/code><\/pre>\n<\/div>\n
What we would expect and would pass our schema validation is this:<\/p>\n
\n
{<\/span><\/span>\r\n    \"metrics\"<\/span>:<\/span> {<\/span><\/span>\r\n        \"labeled_counter\"<\/span>:<\/span> {<\/span><\/span>\r\n            \"glean.validation.pings_submitted\"<\/span>:<\/span> {<\/span><\/span>\r\n                \"events\"<\/span>:<\/span> 1<\/span><\/span>\r\n            }<\/span><\/span>\r\n        },<\/span><\/span>\r\n        ...<\/span><\/span>\r\n    },<\/span><\/span>\r\n    ...<\/span><\/span>\r\n}<\/span><\/span><\/code><\/pre>\n<\/div>\n
The difference? 8 characters:<\/p>\n
\n
-        \"schema: counter\": {<\/span><\/span>\r\n+        \"labeled_counter\": {<\/span><\/span><\/code><\/pre>\n<\/div>\n
8 different characters that still make up valid JSON, but break validation.<\/p>\n
A week later the number of errors kept increasing, affecting up to 2% of all ingested pings from Firefox for Android Beta. That’s worryingly high. That’s enough to drop other work and call an incident.<\/p>\n
The missing schema:_<\/h2>\n
The data we were receiving from a growing number of clients contained 8 bytes that we didn’t expect in that place: schema: <\/code>. That 8-character string didn’t even show up in the Glean SDK source code<\/a>. Where does it come from? Why was it showing up now?<\/p>\n
We did receive entirely valid JSON, so it’s unlikely to be simple memory corruption1<\/sup><\/a>. More like memory confusion, if that’s a thing.<\/p>\n
We know where the payload is constructed. The nested object for labeled metrics is constructed in its own function<\/a>. It starts with string formatting:<\/p>\n
\n
let<\/span> ping_section =<\/span> format!<\/span>(\"labeled_{}\"<\/span>,<\/span> metric.<\/span>ping_section());<\/span><\/span><\/code><\/pre>\n<\/div>\n
My schema? Confused.<\/h2>\n
I don’t know much about how string formatting in Rust works under the hood, but luckily Mara<\/a> blogged about it 2 years ago: Behind the Scenes of Rust String Formatting: format_args!()<\/a> (and then recently improved the implementation<\/a>2<\/sup><\/a>).<\/p>\n
So the format!<\/code> from above expands into something like this:<\/p>\n
\n
std::io::<\/span>_format(<\/span>\r\n    \/\/ Simplified expansion of format_args!():<\/span><\/span>\r\n    std::fmt::<\/span>Arguments {<\/span><\/span>\r\n        template:<\/span> &<\/span>[Str<\/span>(\"labeled_ \"<\/span>),<\/span> Arg(0<\/span>)],<\/span><\/span>\r\n        arguments:<\/span> &<\/span>[&<\/span>metric.<\/span>ping_section() as<\/span> &<\/span>dyn<\/span> Display<\/span>],<\/span><\/span>\r\n    }<\/span><\/span>\r\n);<\/span><\/span><\/code><\/pre>\n<\/div>\n
Another clue that the labeled_<\/code> string is referenced all by itself and swapping out the pointer to it would be enough to lead to the corrupted data we were seeing.<\/p>\n
Architecturing more clues<\/h2>\n
Whenever we’re faced with data anomalies we start by dissecting the data<\/a> to figure out if the anomalies are from a particular subset of clients. The hope is that identifying the subset of clients where it happens gives us more clues about the bug itself.<\/p>\n
After initially focusing too much on actual devices<\/em> colleagues helpfully pointed out that the actual split was the device’s architecture3<\/sup><\/a>:<\/p>\n
\"Data
Data since 2025-11-11 showing a sharp increase in errors for armeabi-v7a clients<\/p><\/div>\n
ARMv8, the 64-bit architecture, did not run into this issue4<\/sup><\/a>. ARMv7, purely 32-bit, was the sole driver of this data anomaly. Another clue that something in the code specifically for this architecture was causing this.<\/p>\n
Logically unchanged<\/h2>\n
With a hypothesis what was happening, but no definite answer why, we went to speculative engineering: Let’s avoid the code path that we think is problematic.<\/p>\n
By explicitly listing out the different strings we want to have in the JSON payload we avoid the formatting and thus hopefully any memory confusion.<\/p>\n
\n
let<\/span> ping_section =<\/span> match<\/span> metric.<\/span>ping_section() {<\/span><\/span>\r\n    \"boolean\"<\/span> =><\/span> \"labeled_boolean\"<\/span>.<\/span>to_string(),<\/span><\/span>\r\n    \"counter\"<\/span> =><\/span> \"labeled_counter\"<\/span>.<\/span>to_string(),<\/span><\/span>\r\n    \/\/ <snip><\/span><\/span>\r\n    _ =><\/span> format!<\/span>(\"labeled_{}\"<\/span>,<\/span> metric.<\/span>ping_section()),<\/span><\/span>\r\n};<\/span><\/span><\/code><\/pre>\n<\/div>\n
\"The
The number of errors have been on a downturn ever since the fix landed on 2025-11-26<\/p><\/div>\n
A bug gone but still there<\/h2>\n
The immediate incident-causing data anomaly was mitigated, the bug is not making it to the Firefox 146 release<\/a>.<\/p>\n
But we still didn’t know why this was happening in the first place. My colleagues Yannis and Serge kept working and searching and were finally able to track down what exactly is happening in the code. The bug<\/a> contains more information on the investigation.<\/p>\n
While I was trying to read and understand the disassembly of the broken builds, they went ahead and wrote a tiny emulator (based on the Unicorn engine<\/a>) that runs just enough of the code to find the offending code path5<\/sup><\/a>.<\/p>\n
> python .\/emulator.py libxul.so\r\nPath: libxul.so\r\nGNU build id: 1b9e9c8f439b649244c7b3acf649d1f33200f441\r\nSymbol server ID: 8F9C9E1B9B43926444C7B3ACF649D1F30\r\nPlease wait, downloading symbols from: https:\/\/symbols.mozilla.org\/try\/libxul.so\/8F9C9E1B9B43926444C7B3ACF649D1F30\/libxul.so.sym\r\nPlease wait, uncompressing symbols...\r\nPlease wait, processing symbols...\r\nProceeding to emulation.\r\nResult of emulation: bytearray(b'schema: ')\r\nThis is a BAD build.<\/code><\/pre>\n
The relevant section of the code boils down to this:<\/p>\n
\n
ldr   r3,<\/span> [<\/span>pc,<\/span> #<\/span>0x20c<\/span>]<\/span><\/span>\r\nadd<\/span>   r3,<\/span> pc<\/span>\r\nstrd  r3,<\/span> r0,<\/span> [<\/span>sp<\/span>,<\/span> #<\/span>0xd0<\/span>]<\/span><\/span>\r\nadd<\/span>   r1,<\/span> sp<\/span>,<\/span> #<\/span>0xd0<\/span><\/span>\r\nbl    alloc::<\/span>fmt::<\/span>format_inner<\/span><\/code><\/pre>\n<\/div>\n
The first two instructions build the pointer to the slice in r3, by using a pc-relative offset found in a nearby constant. Then we store that pointer at sp+0xd0<\/code>, and we put the address sp+0xd0<\/code> into r1<\/code>. So before we reach alloc::fmt::format_inner<\/code>, r1<\/code> points to a stack location that contains a pointer to the slice of interest. The slice lives in .data.rel.ro<\/code> and contains a pointer to the string, and the length of the string (8). The string itself lives in .rodata<\/code>.<\/p><\/blockquote>\n
In good builds the .rodata<\/code> r3<\/code> points to looks like this:<\/p>\n
0x06f0c3d4: 0x005dac18  -->  \"labeled_\"\r\n0x06f0c3d8:        0x8\r\n0x06f0c3dc: 0x0185d707  -->  \"\/builds\/<snip>\/rust\/glean-core\/src\/storage\/mod.rs\"\r\n0x06f0c3e0:       0x4d<\/code><\/pre>\n
In bad builds however it points to something that has our dreaded schema: <\/code> string:<\/p>\n
0x06d651c8: 0x010aa2e8  -->  \"schema: \"\r\n0x06d651cc:        0x8\r\n0x06d651d0: 0x01a869a7  -->  \"maintenance: \"\r\n0x06d651d4:        0xd\r\n0x06d651d8: 0x01a869b4  -->  \"storage dir: \"\r\n0x06d651dc:        0xd\r\n0x06d651e0: 0x01a869c8  -->  \"from variant of type \"\r\n0x06d651e4:       0x15\r\n0x06d651e8: 0x017f793c  -->  \": \"\r\n0x06d651ec:        0x2<\/code><\/pre>\n
This confirms the suspicion that it’s a compiler\/linker bug. Now the question was how to fix that.<\/p>\n
Firefox builds with a variety of Clang\/LLVM versions. Mozilla uses its own build of LLVM and Clang to build the final applications, the exact version used is updated as soon as possible, but never on release<\/a>. Sometimes additional patches are applied on top of the Clang release, like some backports fixing other compiler bugs.<\/p>\n
After identifying that this is indeed a bug in the linker and that it has already been patched in later LLVM versions, Serge did all the work to bisect the LLVM release to find which patches to apply to Mozilla’s own Clang build. Ultimately he tracked it down to these two patches for LLVM:<\/p>\n
This was implemented in 912fc80<\/a> and shipped in Glean v66.1.2<\/a>. It landed in Firefox the same day of the SDK release and made it to Firefox for Android Beta the Friday after. The data shows: It’s working, no more memory confusion!<\/p>\n
There’s our 8-character string labeled_<\/code> that gets swapped. The Glean SDK is embedded into Firefox inside mozilla-central and compiled with all the other code together. A single candidate for the schema: <\/code> string exists in that codebase<\/a>. That’s another clue it could be memory confusion.<\/p>\n
Aside: Telemetry ingestion<\/h2>\n
In Firefox the data is collected using the Glean SDK<\/a>. Data is stored in a local database and eventually assembled into what we call a ping<\/a>: A bundle of related metrics, gathered in a JSON payload to be transmitted. This JSON document is then POST<\/code>ed to the Telemetry edge server<\/a>. From there the decoder eventually picks it up and processes it further. One of the early things it does is verify the received data against one of the pre-defined schemas<\/a>. When data is coming from the Glean SDK it must pass the pre-defined glean.1.schema.json<\/code><\/a>. This essentially describes which fields to expect in the nested JSON object. One thing it is expecting is a labeled_counter<\/code><\/a> A thing it is NOT expecting is schema: counter<\/code>. In fact
\nkeys other than the listed ones are forbidden<\/a>.<\/p>\n