{"id":182,"date":"2020-04-27T19:54:05","date_gmt":"2020-04-27T19:54:05","guid":{"rendered":"https:\/\/blog.mozilla.org\/data\/?p=182"},"modified":"2020-04-27T19:54:05","modified_gmt":"2020-04-27T19:54:05","slug":"this-week-in-glean-glean-for-python-on-windows","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/data\/2020\/04\/27\/this-week-in-glean-glean-for-python-on-windows\/","title":{"rendered":"This Week in Glean: Glean for Python on Windows"},"content":{"rendered":"

(\u201cThis Week in Glean\u201d is a series of blog posts that the Glean Team at Mozilla is using to try to communicate better about our work. They could be release notes, documentation, hopes, dreams, or whatever: so long as it is inspired by Glean. You can find<\/i> an index of all TWiG posts online.<\/i><\/a>)<\/i><\/p>\n

One of the top-line level goals with Glean is to support many different platforms and application types with a single core of code written in Rust.\u00a0 We’ve had Android and iOS support for a while, and are building out the pieces necessary to support desktop Firefox, but today I’d like to talk about how we support Python applications, specifically on Windows.<\/p>\n

Python has a rich C API<\/a> for writing extensions, which are libraries used from Python, but written in compiled languages. This is used by some popular libraries, such as Numpy<\/a>, to achieve higher performance than you could get from Python alone. There are even Rust bindings to this API in PyO3<\/a>, which also helps to handle otherwise manual reference counting automatically.\u00a0 However, for Glean, we don’t use it.<\/p>\n

One of the downsides of using the Python\/C API is that each minor version of Python has its own ABI, so extensions need to be built specially for each version of Python we need to support.\u00a0 In our case, we need to support Python 3.5, 3.6, 3.7 and 3.8.\u00a0 Multiply that be the platforms we need to support (Windows 32-bit, Windows 64-bit, MacOS 64-bit and Linux 64-bit), and that would be 16 separate binaries we would need to build and distribute to support our user base.<\/p>\n

Instead, we compile the Rust code into a standard shared object\/dynamically linked library with a C API. We then use the cffi<\/a> library to interact with it directly from Python.\u00a0 This means we only have to ship one build of Glean for each of the platforms we need to support, not for each version of Python.\u00a0 The shared object is just included as a data file inside the wheel<\/a> that we build, and the cffi library handles opening it and calling methods on it.<\/p>\n

Building Glean for Python on Windows on Linux<\/h3>\n

How we build these wheels for Windows is a bit unusual.\u00a0 We use CircleCI for our continuous testing and building of release products, which doesn’t have Windows support.\u00a0 We could use another CI service with full Windows support, such as AppVeyor.\u00a0 However, adding another service would add additional complexity and cost that isn’t really worth it now, though it might be at some point in the future.<\/p>\n

Fortunately, it’s really easy to build Rust DLLs for Windows on a Linux container.\u00a0 You can see how we do it by looking at our CircleCI configuration<\/a>, and the basic steps are described below.<\/p>\n

First, you need to install the Rust toolchain that targets Windows, for use with the GNU (mingw) toolchain:<\/p>\n

rustup target add x86_64-pc-windows-gnu<\/code><\/p>\n

Then you need to install the mingw toolchain (which is used to get the C standard library and the linker).\u00a0 Since we’re using a Debian VM on CircleCI, we just install the Debian packages:<\/p>\n

sudo apt install -y gcc-mingw-w64<\/code><\/p>\n

We then need to tell Rust’s cargo tool to use this linker when building for Windows by adding the following to the ~\/.cargo\/config<\/code> file:<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0[target.x86_64-pc-windows-gnu]<\/code>\r\n \u00a0\u00a0\u00a0\u00a0 linker = \"\/usr\/bin\/x86_64-w64-mingw32-gcc\u201d<\/code><\/pre>\n
Now we can build Glean the way we normally do, except by passing --target x86_64-pc-windows-gnu<\/code>, it will be built for Windows.<\/p>\n
cargo build --target x86_64-pc-windows-gnu<\/code><\/p>\n
Of course, we also want to test this.\u00a0 In order to do that, we need some way to run the result.\u00a0 For that, we can use Wine<\/a>, which lets you run Windows executables on Linux.\u00a0 It’s easy to install this on Debian:<\/p>\n
sudo apt install wine<\/code><\/p>\n
Through the magic of wine installing a handler for Windows executables, it’s then easy to run Rust unit tests in the regular way, just by telling it to build and test on the Windows target:<\/p>\n
cargo test --target x86_64-pc-windows-gnu<\/code><\/p>\n
That tests the shared object written in Rust.\u00a0 Of course, we should also test that shared object running inside of our Python wrappers.\u00a0 For that, we can install the Windows version of Python inside of Wine.\u00a0 Python for Windows is normally shipped as a Windows installer, but since we’re running this on CircleCI where no one is around to click the buttons in the installer, we instead use the zip file distribution of Python, and extract it.<\/p>\n
wget https:\/\/www.python.org\/ftp\/python\/3.7.7\/python-3.7.7-embed-amd64.zip<\/code>
\nmkdir winpython
\n<\/code>unzip python-3.7.7-embed-amd64.zip -d winpython<\/code><\/p>\n
This distribution of Python doesn’t include pip, so we have to install pip into it before we can install Glean’s dependencies in it.\u00a0 The details are a bit involved, so I refer the reader to our CircleCI config<\/a> for that bit.<\/p>\n
Now we can build a wheel for Glean that contains the Rust code compiled for Windows.\u00a0 This cross-compiling configuration — building compiled code for another target than what you’re building on — is not well-supported by Python’s distutils building infrastructure. So Glean just has a hack: if the GLEAN_PYTHON_MINGW_X86_64_BUILD<\/code> environment variable is set, Glean’s setup.py<\/code> script knows it should build for Windows and put that shared object inside of the wheel (instead of for Linux as it would normally do).\u00a0 Then we hard code the name of the wheel, which pip uses to know what platforms the wheel will work with.<\/p>\n
Once we’ve done that, we can install the Glean wheel inside Python for Windows running on Wine and run all of Glean’s unit tests.<\/p>\n
Stay tuned<\/h3>\n
With those pieces in place, we are able to build wheels to support our Windows users, while still using the Linux-based CI infrastructure we use for the rest of the project.<\/p>\n
In a future installment, I’ll share how we get around some of the multiprocess limitations on Windows by building a super-simple subprocess work manager.<\/p>\n","protected":false},"excerpt":{"rendered":"
(\u201cThis Week in Glean\u201d is a series of blog posts that the Glean Team at Mozilla is using to try to communicate better about our work. They could be release … Read more<\/a><\/p>\n","protected":false},"author":1674,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[315988,448297],"tags":[],"coauthors":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/posts\/182"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/users\/1674"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/comments?post=182"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/posts\/182\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/media?parent=182"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/categories?post=182"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/tags?post=182"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/blog.mozilla.org\/data\/wp-json\/wp\/v2\/coauthors?post=182"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}