(\u201cThis Week in Glean\u201d is a series of blog posts that the Glean Team at Mozilla is using to try to communicate better about our work. They could be release notes, documentation, hopes, dreams, or whatever: so long as it is inspired by Glean.)<\/p>\n
All \u201cThis Week in Glean\u201d blog posts are listed in the TWiG index<\/a>.<\/p>\n <\/p>\n <\/p>\n In the past week Alessio<\/a>, Mike<\/a>, Hamilton<\/a> and I got together for the Glean.js workweek. Our purpose was to build a proof-of-concept of a Glean SDK that works on Javascript environments. You can expect a TWiG in the next few weeks about the outcome of that. Today I am going to talk about something that I tried out in preparation for that week: attempting to compile glean-core to Wasm.<\/p>\n <\/p>\n <\/p>\n <\/p>\n The glean-core<\/a> is the heart of the Glean SDK where most of the logic and functionality of Glean lives. It is written in Rust and communicates with the language bindings in C#, Java, Swift or Python through an FFI layer. For a comprehensive overview of the Glean SDKs architecture, please refer to Jan-Erik\u2019s great blog post<\/a> and talk<\/a> on the subject.<\/p>\n <\/p>\n From the WebAssembly website<\/a>:<\/p>\n \u201cWebAssembly (abbreviated Wasm<\/i>) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable compilation target for programming languages, enabling deployment on the web for client and server applications.\u201d<\/p><\/blockquote>\n Or, from Lin Clark\u2019s \u201cA cartoon intro to WebAssembly\u201d<\/a>:<\/p>\n \u201cWebAssembly is a way of taking code written in programming languages other than JavaScript and running that code in the browser.\u201d<\/p><\/blockquote>\n <\/p>\n <\/p>\n On the Glean team we make an effort to move as much of the logic as possible to glean-core, so that we don\u2019t have too much code duplication on the language bindings and guarantee standardized behaviour throughout all platforms.<\/p>\n Since that is the case, it was counterintuitive for me, that when we set out to build a version of Glean for the web, we wouldn\u2019t rely on the same glean-core as all our other language bindings. The hypothesis was: let\u2019s make JavaScript just another language binding, by making our Rust core compile to a target that runs on the browser.<\/p>\n Rust is notorious for making an effort to have a great Rust to Wasm experience, and the Rust and Webassembly working group<\/a> has built awesome tools that make boilerplate for such projects much leaner.<\/p>\n <\/p>\n <\/p>\n Since this was my first try in doing anything Wasm, I started by following MDN\u2019s guide \u201cCompiling from Rust to WebAssembly\u201d<\/a>, but instead of using their example \u201cHello, World!\u201d Rust project, I used glean-core<\/a>.<\/p>\n From that guide I learned about wasm-pack<\/a>, a tool that deals with the complexities of compiling a Rust crate to Wasm and wasm-bindgen<\/a> a tool that exposes, among many other things, the The first thing that was obvious, was that it would be much harder to try and compile glean-core directly to Wasm. Passing complex types to it has many limitations and I was not able to add the Our FFI crate exposes functions that rely on a global Glean singleton and have relatively simple signatures. These functions are the ones accessed by our language bindings through a C FFI. Most of the Rust complex structures are hidden by this layer from the consumers.<\/p>\n Perfect! I proceeded to add the My assumption that I would be able to just expose the API of Now I got a new error. Progress! If you clone the Glean repository<\/a>, install One of glean-core\u2019s dependencies is rkv<\/a> a storage crate we use for persisting metrics before they are collected and sent in pings. This crate depends on LMDB<\/a> which is written in C, thus the I do not have extensive experience in writing C\/C++ programs, so this was not familiar to me. I figured out that the file this error points to as \u201cnot found\u201d, Internet searching pointed me to wasi-libc<\/a>, a libc for WebAssembly programs. Promising! With this, I retried compiling glean-ffi to Wasm.\u00a0 I just needed to run the build command with added flags:<\/p>\n This didn\u2019t work immediately and the error messages told me to add some extra flags to the command, which I did without thinking much and the final command is:<\/p>\n I would advise the reader now not to get too excited. This command still doesn\u2019t work. It will return yet another set of errors and warnings, mostly related to \u201cusage of undeclared identifiers\u201d or \u201cimplicit declaration of functions\u201d. Most of the identifiers that were erroing started with the \u201cSpecifically, WASI does not yet have an API for creating and managing threads yet, and WASI libc does not yet have pthread support\u201d.<\/p><\/blockquote>\n That was it. I was done with trying to approach the problem of compiling glean-core to Wasm \u201cas is\u201d and I decided to try another way. I could try to abstract away our usage of rkv so that depending on it didn\u2019t block compilation to Wasm, but that is way too big a refactoring task that I considered it a blocker for this experiment.<\/p>\n <\/p>\n <\/p>\n After learning that it would require way too much refactoring of Earlier this year I had a small taste of trying to rewrite part of The histograms code is very self contained so it was a perfect candidate to try and single out for this experiment. I did just that and I was actually able to get it to not error fairly quickly as a standalone thing (you can check out the histogram code<\/a> on the glean-to-wasm repo vs. the histogram code<\/a> on the Glean repo).<\/p>\n After getting this to work I created three accumulation functions that would mimic how each one of the distribution metric types work. These functions would then be exposed to Javascript. The resulting API looks like this:<\/p>\n Each one of these functions creates a histogram, accumulates the given samples to this histogram and returns the resulting histogram as a JSON encoded string. I tried getting them to return For this I was still following MDN\u2019s guide \u201cCompiling from Rust to WebAssembly\u201d<\/a>, which I can\u2019t recommend enough, and after I got my Rust code to compile to Wasm it was fairly straightforward to call the functions imported from the Wasm module inside my Javascript code.<\/p>\n Here is a little taste of what that looked like:<\/p>\n The only hiccup I ran into was that I needed to change my code to use the BigInt<\/a> number type instead of the default Number<\/a> type from Javascript. That is necessary because, in Rust, my functions expect a u64 and BigInt is the type that maps to that from Javascript.<\/p>\n This code can be checked out at: https:\/\/github.com\/brizental\/glean-wasm-experiment<\/a><\/p>\n And there is a demo of it working in: https:\/\/glean-wasm.herokuapp.com\/<\/a><\/p>\n <\/p>\n <\/p>\n This was a very fun experiment, but does it validate my initial hypothesis:<\/p>\n Should we compile glean-core to Wasm and have Javascript be just another language binding?<\/p><\/blockquote>\n We definitely can do that. Even though my first try was not concluded, if we abstract away all the dependencies that we have that can\u2019t be compiled to Wasm, refactor the unsafe functions out and all other possible roadblocks that we find other than these, we can do it. The effort that would take though, I believe is not worth it. It would take us much less time to rewrite glean-core\u2019s code in Javascript. Spoiler alert for our upcoming TWiG about the Glean.js workweek, but in just a week we were able to get a functioning prototype of that.<\/p>\n Our requirements for a Glean software for the web are different from our requirements for a native version of Glean. Different enough that the burden of maintenance for two versions of glean-core, one in Rust and another in Javascript, is probably smaller than the amount of work and hacks it would take to build a single version that attends both platforms.<\/p>\n
\nA quick primer<\/b><\/h2>\n
glean-core<\/b><\/h3>\n
wasm<\/b><\/h3>\n
Why did I decide to do this?<\/b><\/h2>\n
First try: compile glean-core \u201cas is\u201d to Wasm<\/b><\/h2>\n
#[wasm_bindgen]<\/code> attribute which, when added to a function, will make that function accessible from Javascript.<\/p>\n#[wasm_bindgen]<\/code> attribute to trait objects<\/a> or structs that contain trait objects or lifetime annotations. I needed a simpler API surface to make the connection between Rust and Javascript. Fortunately, I had that in hand: glean-ffi<\/code>.<\/p>\n#[wasm_bindgen]<\/code> attribute to one of our entrypoint functions: glean_initialize<\/a>. This uncovered a limitation I didn\u2019t know about: you can\u2019t add this attribute to functions that are unsafe<\/a><\/code>, which unfortunately this one is.<\/p>\nglean-ffi<\/code> to Javascript by compiling it to Wasm without making any changes to it was not holding up. I would have to go through some refactoring to make that work. But until now, I hadn\u2019t gotten to the actual compilation step, the error I was getting was a syntax error. I wanted to go through compilation and see if that completed before diving into any refactoring work. I just removed the\u00a0 #[wasm_bindgen]<\/code> attribute for now and made a new attempt at compiling.<\/p>\nwasm-pack<\/code>, and run wasm-pack build<\/code> inside the glean-core\/ffi\/<\/a><\/code> folder right now, you are bound to get this same error and here is one important excerpt of it:<\/p>\n<...>\r\n\r\nfatal error: 'sys\/types.h' file not found\r\ncargo:warning=#include <sys\/types.h>\r\ncargo:warning= \u00a0 \u00a0 \u00a0\u00a0\u00a0 ^~~~~~~~~~~~~\r\ncargo:warning=1 error generated.\r\nexit code: 1\r\n\r\n--- stderr\r\n\r\nerror occurred: Command \"clang\" \"-Os\" \"-ffunction-sections\" \"-fdata-sections\" \"-fPIC\" \"--target=wasm32-unknown-unknown\" \"-Wall\" \"-Wextra\" \"-DMDB_IDL_LOGN=16\" \"-o\" \"<...>\/target\/wasm32-unknown-unknown\/release\/build\/lmdb-rkv-sys-5e7282bb8d9ba64e\/out\/mdb.o\" \"-c\" \"<...>\/.cargo\/registry\/src\/github.com-1ecc6299db9ec823\/lmdb-rkv-sys-0.11.0\/lmdb\/libraries\/liblmdb\/mdb.c\" with args \"clang\" did not execute successfully (status code exit code: 1)<\/pre>\n
clang<\/code> error.<\/p>\n<sys\/types.h><\/code>, is a header file that should be part of libc. This compiles just fine when trying to compile for our usual targets, so I had a hunch that maybe I just didn\u2019t have the proper libc files for compiling to Wasm targets.<\/p>\nCFLAGS=\"--sysroot=\/path\/to\/the\/newly\/built\/wasi-libc\/sysroot\" wasm-pack build<\/pre>\n
CFLAGS=\"--sysroot=\/path\/to\/wasi-sdk\/clone\/share\/wasi-sysroot -D_WASI_EMULATED_MMAN -D_WASI_EMULATED_SIGNAL\" wasm-pack build<\/pre>\n
pthread_<\/code> prefix, which reminded me of something that I read on the wasi-sdk<\/a>, a toolkit for compiling C programs to WebAssembly that includes wasi-libc,\u00a0 README section:<\/p>\nSecond try: take a part of glean-core and compile that to Wasm<\/b><\/h2>\n
glean-core<\/code> and glean-ffi<\/code> to get them to compile to Wasm, I decided to try a different approach and just get a small self contained part of glean-core<\/code> and compile that to Wasm.<\/p>\nglean-core<\/code> in Javascript for the distribution simulators that we added to The Glean Book<\/a>. To make the simulators work I essentially had to reimplement histograms code and part of the distribution metrics code in Javascript.<\/p>\n#[wasm_bindgen]\r\npub fn accumulate_samples_custom_distribution(\r\n range_min: u32,\r\n range_max: u32,\r\n bucket_count: usize,\r\n histogram_type: i32,\r\n samples: Vec<u64>,\r\n) -> String\r\n\r\n#[wasm_bindgen]\r\npub fn accumulate_samples_timing_distribution(\r\n time_unit: i32,\r\n samples: Vec<u64>\r\n) -> String\r\n\r\n#[wasm_bindgen]\r\npub fn accumulate_samples_memory_distribution(\r\n memory_unit: i32,\r\n samples: Vec<u64>\r\n) -> String\r\n\r\n<\/pre>\n
HashMap<u64,u64><\/code> at first, but that is not supported.<\/p>\nimport(\"glean-wasm\").then(Glean => {\r\n const data = JSON.parse(\r\n Glean.accumulate_samples_memory_distribution(\r\n unit, \/\/ A Number value between 0 - 3\r\n values \/\/ A BigUint64Array with the sample values\r\n )\r\n )\r\n \/\/ <Do something with data<\/code>>\r\n})<\/pre>\nFinal considerations<\/b><\/h2>\n