For the purposes of this story you\u2019ll need to know that there\u2019s two times when Glean notices the product\u2019s gone from \u201cdata upload: on\u201d to \u201cdata upload: off\u201d: while Glean is running, and during Glean startup. If Glean\u2019s running, then we just handle things \u2013 we were told the setting changed from \u201cdata upload: on\u201d to \u201cdata upload: off\u201d and away we go. But Glean knows that it isn\u2019t always listening to the data upload setting, so if it it starts up with \u201cdata upload: off\u201d and the last time it shut down we were \u201cdata upload: on\u201d we\u2019ll send a specific \u201cat_init\u201d-reason \u201cdeletion-request\u201d ping.<\/p>\r\n
We in the Data Org monitor how Glean is behaving. One thing we\u2019ve learned about how Glean behaves is that the number of \u201cdeletion-request\u201d pings is roughly constant over time. And the proportion of \u201cdeletion-request\u201d pings that have the \u201cat_init\u201d reason should remain a fairly fixed one.<\/p>\r\n
What shouldn\u2019t happen is for Firefox Desktop-sent \u201cat_init\u201d-reason \u201cdeletion-request\u201d pings to spike like this on January 5:<\/p>\r\n
<\/p>\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n <\/p>\r\n\r\n What we do when we notice things like this is file a bug<\/a>. As the one responsible for Glean\u2019s integration in Firefox Desktop, and as someone with a long history of looking into anomalies<\/a>, I took a look. At this initial point I was pretty sure it\u2019d be a single actor (a single user, a single company, a single internet cafe) doing something odd\u2026 but alas, the evidence was inconclusive:<\/p>\r\n Evidence consistent with a single actor being responsible for it all:<\/p>\r\n Evidence inconsistent with a single actor being responsible for it all:<\/p>\r\n Regardless of why it was happening, it quickly became more important that we learn what we needed to do about it. We spun up an Incident, which is how we organize ourselves when there\u2019s something happening that requires cross-functional collaboration and isn\u2019t getting better on its own. Once there we ascertained that we could respond very quickly and decisively and do<\/p>\r\n Nothing at all.<\/p>\r\n The volume of these pings vastly eclipsed any other \u201cdeletion-request\u201d pings we would otherwise have received, so you\u2019d be forgiven for thinking that it was terribly expensive to receive, store, and process them all. In reality, we batch these requests. And even before this spike, every batch of requests required editing every partition of every table. Adding another list of identifiers to delete equal in size to two times the peak Firefox Desktop population in Korea just doesn\u2019t matter all that much.<\/p>\r\n The pressure was off. Even if it got worse\u2026 which it did:<\/p>\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n <\/p>\r\n\r\n On March 26, when it reached and maintained a peak of five times the volume of the Firefox Desktop population in Korea, it still wasn\u2019t harming our data platform\u2019s ability to serve business needs or costing us all that much in operational spend. We didn\u2019t need to invest effort into running down the source, so we didn\u2019t.<\/p>\r\n And so I just kept an occasional eye on it until, just as suddenly but not quite as abruptly as it began, on April 12 the ping volumes began to decrease. By April 18, we were back to normal levels.<\/p>\r\n\r\n\r\n\r\n\r\n\r\n <\/p>\r\n\r\n We had successfully ignored it until it went away.<\/p>\r\n So what happened to Korean Firefox Desktop users from Jan 5 to April 12, 2023? We never figured it out. If you know about something happening across those dates in Korea: please get in touch<\/a>. As little as it needed solving for the sake of business needs, it still needs solving for the sake of my curiosity.<\/p>\r\n :chutten<\/p>\r\n (( This is a syndicated copy of the original post<\/a>. ))<\/p>\r\n\r\n\r\n\r\n\r\n","protected":false},"excerpt":{"rendered":" Something happened on January 5, 2023. All of a sudden we abruptly started receiving a number of pings from Firefox Desktop clients in Korea equal to two times the size … Read more<\/a><\/p>\n","protected":false},"author":1437,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[315988,323282],"tags":[525,30],"coauthors":[311808],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/posts\/428"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/users\/1437"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/comments?post=428"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/posts\/428\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/media?parent=428"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/categories?post=428"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/tags?post=428"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/coauthors?post=428"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"time-series](\"https:\/\/chuttenblog.files.wordpress.com\/2023\/04\/image.png\")
<\/figure>\r\n\r\n\r\n
\r\n
![\"Time-series](\"https:\/\/chuttenblog.files.wordpress.com\/2023\/04\/image-1.png\")
<\/figure>\r\n\r\n![\"Time-series](\"https:\/\/chuttenblog.files.wordpress.com\/2023\/04\/image-2.png?w=746\")
<\/figure>\r\n\r\n