(\u201cThis Week in Glean\u201d is a series of blog posts that the Glean Team at Mozilla is using to try to communicate better about our work. They could be release notes, documentation, hopes, dreams, or whatever: so long as it is inspired by Glean.)<\/p>\n
Last week\u2019s blog post:This Week in Glean: mozregression telemetry (part 2)<\/a> by William Lachance.<\/p>\n All \u201cThis Week in Glean\u201d blog posts are listed in the TWiG index<\/a> \u00a0(and on the Mozilla Data blog<\/a>).<\/p>\n In the Glean SDK, when a ping is submitted<\/a> it gets internally persisted to disk and then queued for upload. The actual upload may happen later on, depending on factors such as the availability of an Internet connection or throttling. To save users\u2019 bandwidth and reduce the costs to move bytes within our pipeline, we recently introduced gzip<\/a> compression for outgoing pings.<\/p>\n This article will go through some details of our upload system and what it took us to enable the ping compression.<\/p>\n How does ping uploading work?<\/strong><\/p>\n Within the Glean SDK, the glean-core Rust component does not provide any specific implementation to perform the upload of pings<\/a>. This means that either the language bindings (e.g. Glean APIs for Android in Kotlin) or the product itself (e.g. Fenix) have to provide a way to transport data from the client to the telemetry endpoint.<\/p>\n Before our recent changes<\/a> (by Beatriz Rizental and Jan-Erik) to the ping upload system, the language bindings needed to understand the format with which pings were persisted to disk in order to read and finally upload them. This is not the case anymore: glean-core will provide language bindings with the headers and the data (ping payload!) of the request they need to upload.<\/p>\n The new upload API empowers the SDK to provide a single place in which to compress the payload to be uploaded: glean-core, right before serving upload requests to the language bindings.<\/p>\n gzipping: the implementation details<\/strong><\/p>\n The implementation<\/a> of the function to compress the payload is trivial, thanks to the `flate2` Rust crate:<\/p>\n And an even simpler way<\/a> to use it to compress the body of outgoing requests:<\/p>\n What\u2019s next?<\/strong><\/p>\n The new upload mechanism and its compression improvement is only currently available for the iOS and Android Glean SDK language bindings. Our next step (currently in progress<\/a>!) is to add the newer APIs to the Python bindings as well, moving the complexity of handling the upload process to the shared Rust core.<\/p>\n In our future, the new upload mechanism will additionally provide a flexible constraint-based scheduler (e.g. \u201csend at most 10 pings per hour\u201d) in addition to pre-defined rules for products to use.<\/p>\n","protected":false},"excerpt":{"rendered":" (\u201cThis Week in Glean\u201d is a series of blog posts that the Glean Team at Mozilla is using to try to communicate better about our work. They could be release … Read more<\/a><\/p>\n","protected":false},"author":1538,"featured_media":197,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[448297],"coauthors":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/posts\/196"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/users\/1538"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/comments?post=196"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/posts\/196\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/media\/197"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/media?parent=196"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/categories?post=196"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/tags?post=196"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/coauthors?post=196"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\/\/\/ Attempt to gzip the provided ping content.
\nfn gzip_content(path: &str, content: &[u8]) -> Option<Vec<u8>> {
\nlet mut gzipper = GzEncoder::new(Vec::new(), Compression::default());
\n\/\/ Attempt to add the content to the gzipper.
\nif let Err(e) = gzipper.write_all(content) {
\nlog::error!(\"Failed to write to the gzipper: {} - {:?}\", path, e);
\nreturn None;
\n}
\ngzipper.finish().ok()
\n}<\/code><\/p><\/blockquote>\npub fn new(document_id: &str, path: &str, body: JsonValue) -> Self {
\nlet original_as_string = body.to_string();
\nlet gzipped_content = Self::gzip_content(path, original_as_string.as_bytes());
\nlet add_gzip_header = gzipped_content.is_some();
\nlet body = gzipped_content.unwrap_or_else(|| original_as_string.into_bytes());
\nlet body_len = body.len();
\nSelf {
\ndocument_id: document_id.into(),
\npath: path.into(),
\nbody,
\nheaders: Self::create_request_headers(add_gzip_header, body_len),
\n}
\n}<\/code><\/p><\/blockquote>\n