Note: All times are based on AMO being compressed on a MacBook. On our build server, those 80 seconds took about 160 seconds. Most times are averages and/or estimates, and are based on an initial 80 seconds.
UglifyJS and clean-css (-43s)
Clean-css works much like YUI (lots of regex), which is a bit unfortunate. However, it does so much faster — 10x as fast, to be exact. [CSS Benchmark results]
Original Time: 80s
New Time: 37s
Saving File Hashes (-2s)
We were fetching the hash (originally the git commit id, now hash of the actual file) for the same files fairly often. By sticking the already-computed hashes into a dict, we were able to shave off a small but significant amount of time.
Original Time: 37s
New Time: 35s
Use File Hashes For Image Cache Busting (-12s)
When I first wrote cache busting for individual images in our CSS files, I used the image’s most recent git commit as the cache-busting hash. As it turns out, your script gets really slow when you’re looking up the history for dozens of images using GitPython. So, I switched to using a hash of the file. The end result is the same — the hash is only updated if the image is changed. However, using file hashes instead of git commits shaved a good chunk of time off our build process.
Original Time: 35s
New Time: 23s
Only Minify When Necessary (up to -17s)
I got down to 23 seconds, and was stuck. The biggest time sink in the process was the minifying of files, which I couldn’t do much about without touching the UglifyJS or clean-css code. Even if I wanted to, I doubted I could do much to speed either of them up. After thinking about it for a bit, I realized that there was no reason for us to be minifying every file each time. We only needed to minify the files that had actually changed. So, jingo-minify now writes the concatted file out to a temporary location, and compares it to the preexisting concatted file. If they’re the same file, we have no need to re-minify.
Since we push to preview every time anyone commits and most commits don’t modify more than one or two JS or CSS files (if any), we can save between 15 and 17 seconds each time.
Original Time: 23s
New Time: 6s
So, that’s how I managed to speed up our asset compression by over an order of magnitude — from 80 seconds down to about 6. (Note: Time is based on the script on a MacBook. On the slower build server, it’s closer to 160 seconds down to 9.) Hopefully this means much smoother pushes for AMO, SUMO and anyone else using jingo-minify to compress their assets.