{"id":202,"date":"2020-06-12T20:28:42","date_gmt":"2020-06-12T20:28:42","guid":{"rendered":"https:\/\/blog.mozilla.org\/data\/?p=202"},"modified":"2020-06-12T20:28:42","modified_gmt":"2020-06-12T20:28:42","slug":"this-week-in-glean-project-fog-update-end-of-h12020","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/data\/2020\/06\/12\/this-week-in-glean-project-fog-update-end-of-h12020\/","title":{"rendered":"This Week in Glean: Project FOG Update, end of H12020"},"content":{"rendered":"

(\u201cThis Week in Glean\u201d is a series of blog posts that the Glean Team at Mozilla is using to try to communicate better about our work. They could be release notes, documentation, hopes, dreams, or whatever: so long as it is inspired by Glean. You can find an index of all TWiG posts online<\/a>.)<\/p>\n

It\u2019s been a while since last I wrote on Project FOG, so I figure I should update all of you on the progress we\u2019ve made.<\/p>\n

A reminder: Project FOG (Firefox on Glean)<\/a> is the year-long effort to bring the Glean SDK to Firefox. This means answering such varied questions as \u201cWhere are the docs going to live?\u201d (here<\/a>) \u201cHow do we update the SDK when we need to?\u201d (this way<\/a>) \u201cHow are tests gonna work?\u201d (with difficulty<\/a>) and so forth. In a project this long you can expect updates from time-to-time. So where are we?<\/p>\n

First, we\u2019ve added the Glean SDK to Firefox Desktop and include it in Firefox Nightly<\/a>. This is only a partial integration, though, so the only builtin ping it sends is the \u201cdeletion-request\u201d ping when the user opts out of data collection in the Preferences. We don\u2019t actually collect any data, so the ping doesn\u2019t do anything, but we\u2019re sending it and soon we\u2019ll have a test ensuring that we keep sending it<\/a>. So that\u2019s nice.<\/p>\n

Second, we\u2019ve written a lot of Design Proposals. The Glean Team and all the other teams our work impacts are widely distributed across a non-trivial fragment of the globe<\/a>. To work together and not step on each others\u2019 toes we have a culture of putting most things larger than a bugfix into Proposal Documents<\/a> which we then pass around asynchronously for ideation, feedback, review, and signoff. For something the size and scope of adding a data collection library to Firefox Desktop, we\u2019ve needed more than one<\/a>. These design proposals are Google Docs for now, but will evolve to in-tree documentation (like this<\/a>) as the proposals become code. This way the docs live with the code and hopefully remain up-to-date for our users (product developers, data engineers, data scientists, and other data consumers), and<\/em> are made open to anyone in the community who\u2019s interested in learning how it all works.<\/p>\n

Third, we have a Glean SDK Rust API! Sorta. To limit scope creep we haven\u2019t added the Rust API to mozilla\/glean<\/a> and are testing its suitability in FOG itself. This allows us to move a little faster by mixing our IPC<\/a> implementation directly into the API, at the expense of needing to extract the common foundation later. But when we do extract it, it will be fully-formed and ready for consumers since it\u2019ll already have been serving the demanding needs of FOG.<\/p>\n

Fourth, we have tests. This was a bit of a struggle as the build order of Firefox means that any Rust code we write that touches Firefox internals can\u2019t be tested in Rust tests<\/a> (they must be tested by higher-level integration tests instead). By damming off the Firefox-adjacent pieces of the code we\u2019ve been able to write and run Rust tests of the metrics API after all. Our code coverage is still a little low, but it\u2019s better than it was.<\/p>\n

Fifth, we are using Firefox\u2019s own network stack to send pings. In a stroke of good fortune the application-services team (responsible for fan-favourite Firefox features \u201cSync<\/a>\u201d, \u201cSend Tab<\/a>\u201d, and \u201cFirefox Accounts<\/a>\u201d) was bringing a straightforward Rust networking API called Viaduct<\/a> to Firefox Desktop almost exactly when we found ourselves in need of one. Plugging into Viaduct was a breeze<\/a>, and now our \u201cdeletion-request\u201d pings<\/a> can correctly work their way through all the various proxies and protocols to get to Mozilla\u2019s servers.<\/p>\n

Sixth, we have firm designs on how to implement both the C++ and JS APIs in Firefox. They won\u2019t be fully-fledged language bindings the way that Kotlin, Python, and Swift<\/a> are (( they\u2019ll be built atop the Rust language binding so they\u2019re really more like shims )), but they need to have every metric type and every metric instance that a full language binding would have, so it\u2019s no small amount of work.<\/p>\n

But where does that leave our data consumers? For now, sadly, there\u2019s little to report on both the input and output sides: We have no way for product engineers to collect data in Firefox Desktop (and no pings to send the data on), and we have no support in the pipeline for receiving data, not that we have any to analyse. These will be coming soon, and when they do we\u2019ll start cautiously reaching out to potential first customers to see whether their needs can be satisfied by the pieces we\u2019ve built so far.<\/p>\n

And after that? Well, we need to do some validation work to ensure we\u2019re doing things properly. We need to implement the designs we proposed. We need to establish how tasks accomplished in Telemetry can now be accomplished in the Glean SDK. We need to start building and shipping FOG and the Glean SDK beyond Nightly to Beta and Release. We need to implement the builtin Glean SDK pings. We need to document the designs so others can understand them, best practices so our users can follow them, APIs so engineers can use them, test guarantees so QA can validate them, and grand processes for migration from Telemetry to Glean so that organizations can start roadmapping their conversions.<\/p>\n

In short: plenty has been done, and there\u2019s still plenty to do.<\/p>\n

I guess we\u2019d better be about it, then.<\/p>\n

:chutten<\/p>\n

(( this is a syndicated copy of the original post<\/a> ))<\/p>\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

(\u201cThis Week in Glean\u201d is a series of blog posts that the Glean Team at Mozilla is using to try to communicate better about our work. They could be release … Read more<\/a><\/p>\n","protected":false},"author":1437,"featured_media":197,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[448297],"tags":[30,448307,448297],"coauthors":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/posts\/202"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/users\/1437"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/comments?post=202"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/posts\/202\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/media\/197"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/media?parent=202"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/categories?post=202"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/tags?post=202"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/coauthors?post=202"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}