{"id":134,"date":"2012-01-19T17:21:33","date_gmt":"2012-01-20T01:21:33","guid":{"rendered":"http:\/\/blog.mozilla.org\/warner\/?p=134"},"modified":"2012-01-19T17:21:33","modified_gmt":"2012-01-20T01:21:33","slug":"improving-the-pynacl-build-process","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/warner\/2012\/01\/19\/improving-the-pynacl-build-process\/","title":{"rendered":"Improving the pynacl build process"},"content":{"rendered":"<p>I&#8217;ve been hacking on <a href=\"https:\/\/github.com\/warner\/pynacl\">my copy of pynacl<\/a> this week. pynacl is a set of Python bindings to the <a href=\"http:\/\/nacl.cr.yp.to\/\">NaCl cryptography library<\/a> (by djb and friends). The bindings, first written by <a href=\"https:\/\/github.com\/seanlynch\/pynacl\">Sean Lynch<\/a> and then picked up by <a href=\"https:\/\/github.com\/k3d3\/pynacl\">&#8220;k3d3&#8221;<\/a>, are great. They even support py2.6 through py3.2.<\/p>\n<p>But actually building them is a hassle, because the NaCl build process is so idiosyncratic. It consists of a 500-line undocumented shell script named &#8220;do&#8221;, and running it gets you 25 minutes of 100% CPU that executes in stony silence (all progress messages are redirected to a logfile). If you can wait that long, and think to explore the directory afterwards, you&#8217;ll be rewarded with a build\/HOSTNAME\/ directory that contains a libnacl.a and a set of header files that are pretty easy to use. What&#8217;s actually going on behind the scenes is that the script is exhaustively compiling and testing a large matrix of compiler flags (-O vs -O3 vs -O3 -funroll-loops), ABI variants, and alternative implementations. The goal is apparently to:<\/p>\n<ul>\n<li>select the fastest possible implementation and compiler options, using any assembly-language tricks specific to the processor (SSE3, etc)<\/li>\n<li>make sure the unit tests pass<\/li>\n<li>construct a performance report to send back to the authors<\/li>\n<\/ul>\n<p>Unfortunately, this doesn&#8217;t play well with other build systems that might want to embed a copy, such as Python&#8217;s distutils, because:<\/p>\n<ul>\n<li>some compiler flags (-fPIC) are needed to build the .so files that python can load at runtime: distutils knows what these are, &#8220;do&#8221; doesn&#8217;t<\/li>\n<li>when building e.g. debian packages, the results will be used on other machines, so processor-specific optimizations aren&#8217;t ok (you might build your packages on a machine with some feature that&#8217;s not present on the machines that use those packages, so stick with least-common-denominator).<\/li>\n<li>running &#8220;do&#8221; requires a Bourne Shell interpreter, standard on unix systems but not so obvious on windows. distutils knows how to compile things on windows, but you have to tell it the source files, and it will run the compiler itself.<\/li>\n<li>having a separate &#8220;.\/do&#8221; compile step means that &#8220;setup.py build&#8221; is not enough, which means that easy_install won&#8217;t work, making it hard to use pynacl as a dependency in virtualenv or pip environments.<\/li>\n<\/ul>\n<p>In addition, waiting 25 minutes for an otherwise small and elegant library to build is just a drag.<\/p>\n<p><!--more--><\/p>\n<p>So I&#8217;ve pursued two projects this week. The first is to simply embed a mostly-unmodified (I did add -fPIC) copy of the latest nacl release into the pynacl source tree, and modify pynacl&#8217;s setup.py to use it. The build process becomes &#8220;cd nacl-*; .\/do; cd ..; python setup.py build&#8221;. This is way easier than downloading+building an external copy and then pointing environment variables at the result. Takes just as long, but is easier to run. This work is in my <a href=\"https:\/\/github.com\/warner\/pynacl\/tree\/embed-nacl\">&#8220;embed-nacl&#8221;<\/a> branch (don&#8217;t be surprised if the branch is gone by the time you read this.. it may get merged to trunk). I also have a subset of &#8220;do&#8221;, named &#8220;dont&#8221;, which cuts out some of the unused pieces (like the C++ bindings and the large performance benchmarks), and runs in about 8 minutes.<\/p>\n<p>The second is to embed a modified subset of the nacl sources in pynacl, making it look much more like a normal python extension module. This is in my <a href=\"https:\/\/github.com\/warner\/pynacl\/tree\/minimal\">&#8220;minimal&#8221;<\/a> branch (again, this branch may get merged and be deleted).<\/p>\n<p>This turned out to be pretty hard: the nacl &#8220;do&#8221; script synthesizes different versions of the same header file for each algorithm, and deletes many of them just after compilation. Also, the C code for e.g. crypto_hash_sha256 uses the same function name crypto_hash() as the code for crypto_hash_sha512, and depends upon #defines in the header file to prevent them from colliding. (the goal here is to let users call short function names and get the recommended algorithm, like crypto_box() instead of crypto_box_curve25519xsalsa20poly1305(), which is frequently a good idea, but not always).<\/p>\n<p>So I had to modify the .c files to have non-colliding names, and write new header files to provide the recommended-algorithm mapping. I only copied in the portable implementations (leaving out the non-portable asm speedups). I also left out the multiple-ABI support, the try-different-compiler-flags speedups, the automatic performance tests, and the unit tests (although I hope to bring those back). But the result is something that compiles in <strong>13\u00a0seconds<\/strong>, a 100x speedup, and which can be built with easy_install.<\/p>\n<p>How does performance suffer without the specialized implementations and compiler flags? Here&#8217;s a comparison of the time it takes to call the various python functions (using 1000-byte messages, where applicable) in the two versions, on my 2010 MacBookPro (2.8GHz Core2Duo). The results are tolerable. The biggest victim is the Curve25519 scalar multiplication functions, since the whole algorithm was specifically designed to take advantage of 64-bit operations not available to the generic version (which needs to run on 32-bit machines). This slowdown also hits the high-level box()\/box_open() functions, bringing them up to 5ms on my laptop. crypto_sign uses an early version of Ed25519, and nacl does not yet have an optimized version, so it takes 6ms in both branches. The next version of nacl will include the same carefully optimized version of Ed25519 as the SUPERCOP benchmark suite, in which signing takes about 19us, so this will also become a noticeable slowdown.<\/p>\n<table style=\"border: 1px solid #CCC;\" cellspacing=\"0\">\n<tbody>\n<tr>\n<th style=\"padding-left: 10px; padding-right: 10px;\">name<\/th>\n<th style=\"padding-left: 10px; padding-right: 10px;\">ref<\/th>\n<th style=\"padding-left: 10px; padding-right: 10px;\">optimized<\/th>\n<th style=\"padding-left: 10px; padding-right: 10px;\">slowdown<\/th>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid #CCC; text-align: left;\">crypto_auth<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">8.50 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">7.95 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right; background-color: #afa;\">1.1x<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid #CCC; text-align: left;\">crypto_auth_verify<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">8.49 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">7.87 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right; background-color: #afa;\">1.1x<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid #CCC; text-align: left;\">crypto_box<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">4.64 ms<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">198.61 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right; background-color: #faa;\">23.3x<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid #CCC; text-align: left;\">crypto_box_keypair<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">4.57 ms<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">198.84 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right; background-color: #faa;\">23.0x<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid #CCC; text-align: left;\">crypto_box_open<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">4.65 ms<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">199.33 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right; background-color: #faa;\">23.3x<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid #CCC; text-align: left;\">crypto_hash_sha256<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">898.93 ns<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">855.99 ns<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right; background-color: #afa;\">1.1x<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid #CCC; text-align: left;\">crypto_hash_sha512<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">1.54 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">1.25 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right; background-color: #afa;\">1.2x<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid #CCC; text-align: left;\">crypto_onetimeauth<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">49.06 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">2.40 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right; background-color: #faa;\">20.4x<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid #CCC; text-align: left;\">crypto_onetimeauth_verify<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">67.07 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">2.42 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right; background-color: #faa;\">27.7x<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid #CCC; text-align: left;\">crypto_scalarmult<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">4.57 ms<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">191.88 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right; background-color: #faa;\">23.8x<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid #CCC; text-align: left;\">crypto_scalarmult_base<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">4.69 ms<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">192.76 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right; background-color: #faa;\">24.3x<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid #CCC; text-align: left;\">crypto_secretbox<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">58.02 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">5.09 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right; background-color: #faa;\">11.4x<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid #CCC; text-align: left;\">crypto_secretbox_open<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">58.59 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">5.64 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right; background-color: #faa;\">10.4x<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid #CCC; text-align: left;\">crypto_sign<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">6.28 ms<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">5.95 ms<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right; background-color: #afa;\">1.1x<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid #CCC; text-align: left;\">crypto_sign_keypair_fromseed<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">6.26 ms<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">5.96 ms<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right; background-color: #afa;\">1.1x<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid #CCC; text-align: left;\">crypto_sign_open<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">16.58 ms<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">15.67 ms<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right; background-color: #afa;\">1.1x<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid #CCC; text-align: left;\">crypto_stream<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">5.53 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">2.66 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right; background-color: #ffa;\">2.1x<\/td>\n<\/tr>\n<tr>\n<td style=\"border: 1px solid #CCC; text-align: left;\">crypto_stream_xor<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">6.80 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right;\">2.27 us<\/td>\n<td style=\"border: 1px solid #CCC; text-align: right; background-color: #ffa;\">3.0x<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>But I think the result is still pretty good: for many applications where you&#8217;re using Python, 5ms is just as good as 200us. And this branch makes it <strong>much<\/strong> easier to depend upon pynacl than the others.<\/p>\n<p>My next steps are to clean up the branch a bit, talk to k3d3 about which approach seems the best, get it merged upstream, and then hopefully get it copied to PyPI. And then investigate how hard it&#8217;d be to build the alternative implementations: maybe with some distutils hacking, we could build all the variants (and see which ones actually work), do some quick performance tests, then call setup() again &#8220;for real&#8221; with just the fastest ones. That should shave off most of the slowdown (leaving the compiler-flag gains), but still give us something that compiles easily.<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I&#8217;ve been hacking on my copy of pynacl this week. pynacl is a set of Python bindings to the NaCl cryptography library (by djb and friends). The bindings, first written by Sean Lynch and then picked up by &#8220;k3d3&#8221;, are great. They even support py2.6 through py3.2. But actually building them is a hassle, because [&hellip;]<\/p>\n","protected":false},"author":328,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10479],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/warner\/wp-json\/wp\/v2\/posts\/134"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/warner\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/warner\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/warner\/wp-json\/wp\/v2\/users\/328"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/warner\/wp-json\/wp\/v2\/comments?post=134"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/warner\/wp-json\/wp\/v2\/posts\/134\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/warner\/wp-json\/wp\/v2\/media?parent=134"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/warner\/wp-json\/wp\/v2\/categories?post=134"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/warner\/wp-json\/wp\/v2\/tags?post=134"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}