Extravaganza – December 2015

Once a month, web developers from across Mozilla get together to prepare free SSL tickets to hand out at the Let’s Encrypt SSL Kitchen. While we train volunteers to serve hot SSL tickets to needy websites, we find time to talk about the work that we’ve shipped, share the libraries we’re working on, meet new folks, and talk about whatever else is on our minds. It’s the Webdev Extravaganza! The meeting is open to the public; you should stop by!

You can check out the wiki page that we use to organize the meeting, or view a recording of the meeting in Air Mozilla. Or just read on for a summary!

Shipping Celebration

The shipping celebration is for anything we finished and deployed in the past month, whether it be a brand new site, an upgrade to an existing one, or even a release of a library.

donate.mozilla.org Shipped!

First off was jbuck with the exciting news that the rebuilt donate.mozilla.org is live and processing donations! Previously hosted by vendor Blue State Digital, the new page uses PayPal and Stripe to handle payments. Since launching, the new page has already processed over $1 million in donations!

Pip 8 Hashing Support

Next was ErikRose, who shared the news that pip 8 will support hash verification, nearly obsoleting the need for peep! Exciting new features:

  • New official syntax for specifying hashes
  • Hexadecimal hashes for generating hashes with other programs like OpenSSL
  • Hashes for wheels
  • Better error messaging and unfulfilled dependency detection
  • HTTP caching support

Pip 8 is expected to be released sometime in January.

MDN Compatibility Tables

Shobson stopped by to mention that a new design for compatibility tables on MDN has been enabled for beta users on select pages. The new design is inspired by CanIUse and, in addition to being easier to read, are also mobile-friendly. The new design is the first public feature using MDN’s in-development compatibility API.

DXR Large File Performance Improvements

Back to ErikRose, with news of performance improvements on DXR for pages that have many links on them. After profiling, Erik found that pages that generate many URLs spend a majority of their time building and escaping URLs due to Werkzeug‘s custom URL escaping. By switching to his own hard-coded URL generation functions, the load time on the affect pages improved 4x from 17 seconds down to 3 seconds. Hooray!

Roundtable

The Roundtable is the home for discussions that don’t fit anywhere else.

Starfield Charts

ErikRose shared an interesting story about trying to understand logs from Firefox Hello. The logs available to him showed anonymous sessions with join and leave events, and dots to represent time segments where nothing happened. After sorting these logs by length and zooming out his text editor, Erik was able to notice patterns in the data thanks to his editor’s anti-aliasing of the zoomed-out text. Hooray for poor-man’s data analysis!

Sched Colorblind User Style for Stylish

Next was shobson again, with a neat little fix for colorblind people using Sched, the scheduling service Mozilla uses for its workweeks. She created a Stylish userstyle that adds colored stripes to the sides of event boxes on Sched using colors that are visually differentiable to the most common forms of colorblindness. If you’re colorblind and are having trouble using Sched, install Stylish and try the stylesheet out!

Mozlando Recommended Sessions

For Mozillians planning to attend the Mozlando workweek, several people brought up recommended sessions to attend:


Along with the SSL Kitchen, we also volunteer at the MDN job training school to help websites get the skills they need to follow web standards and break the cycle of -webkit-ness for good!

If you’re interested in web development at Mozilla, or want to attend next month’s Extravaganza, subscribe to the dev-webdev@lists.mozilla.org mailing list to be notified of the next meeting, and maybe send a message introducing yourself. We’d love to meet you!

See you next month!

Beer and Tell – November 2015

Once a month, web developers from across the Mozilla Project get together to design programming languages that are intentionally difficult to reason about. While we advanced the state-of-the-art in side effects, we find time to talk about our side projects and drink, an occurrence we like to call “Beer and Tell”.

There’s a wiki page available with a list of the presenters, as well as links to their presentation materials. There’s also a recording available courtesy of Air Mozilla.

Peterbe: Headsupper.io

Peterbe started us off with headsupper.io, a service that sends notification emails when you commit to a GitHub project with a specific keyword in your commit message. The service is registered as a Github webhook, and you can configure the service to only send emails on new tags if you so desire.

Osmose: Advanced Open File (Round 2)

Next up was Osmose (that’s me!), with an Atom package for opening files called Advanced Open File. Advanced Open File adds a convenient modal dialog for finding files to open or create that aims to replace use of the system file dialog. Previously featured on Beer and Tell, today’s update included news of a rewrite in ES2015 using Babel, test coverage, Windows path fixes, and more!

Kumar: React + Redux Live Reload

Kumar shared a demo of an impressive React and Redux development setup that includes live-reloading of the app as the code changes, as well as a detailed view of the state changes happening in the app and the ability to walk through the history of state changes to debug your app. The tools even replay state changes after live-reloading for an impressively short feedback loop during development.

Bwalker: ebird-mybird

Bwalker was next with a site called ebird-mybird. eBird is a bird observation checklist that bird watchers can use to track their observations. ebird-mybird reads in a CSV file exported from eBird and displays the data in various useful forms on a static site, including aggregate sightings by year/month and sightings categorized by species, location, and date.

The site itself is a frontend app that uses C3 for generating charts, PapaParse for parsing the CSV files, and Handlebars for templating.

Potch: Pseudorandom Number Generator

Last up was Potch with a small experiment in generating pseudorandom numbers in JavaScript. Inspired by a blog post about issues with Math.random in V8, Potch create a very simple Codepen that draws on a canvas based on custom-generated random numbers.

If you need sound random number generation, the blog post recommends crypto.randomBytes, also included in the Node standard library.


This week’s result was a programming language composed entirely of pop culture references, including a time-sensitive compiler that assigns optimization levels based on how current your references are.

If you’re interested in attending the next Beer and Tell, sign up for the dev-webdev@lists.mozilla.org mailing list. An email is sent out a week beforehand with connection details. You could even add yourself to the wiki and show off your side-project!

See you next month!

Extravaganza – November 2015

Once a month, web developers from across Mozilla get together to revolutionize layoffs. While we calculate the optimum distance away an employee should be from their desk when they’re informed of their impending doom, we find time to talk about the work that we’ve shipped, share the libraries we’re working on, meet new folks, and talk about whatever else is on our minds. It’s the Webdev Extravaganza! The meeting is open to the public; you should stop by!

You can check out the wiki page that we use to organize the meeting, or view a recording of the meeting in Air Mozilla. Or just read on for a summary!

Shipping Celebration

The shipping celebration is for anything we finished and deployed in the past month, whether it be a brand new site, an upgrade to an existing one, or even a release of a library.

Pontoon is replacing Verbatim

Osmose (that’s me!) started us off with the news that Pontoon, a Mozilla-developed website for translating software, is going to replace Verbatim, Mozilla’s instance of Pootle, which is an externally-developed website for translating software. Pontoon is able to do certain things that Pootle doesn’t, such as in-page translation.

Over 60% of active localization teams have voluntarily moved to Pontoon already, and the team currently aims to have 100% of active localization to have moved over by the first week of December.

mozilla.org on Deis Update

Next was jgmize, who shared a small update about the previously-mentioned move of mozilla.org to Deis, an open-source PaaS. The site has now fully transitioned out of the PHX1 datacenter and is split between Deis on AWS and SCL3.

Open-source Citizenship

Here we talk about libraries we’re maintaining and what, if anything, we need help with for them.

django-browserid 1.0.1

Last was Osmose again with the news that django-browserid 1.0.1 has been released. The only changes are the addition of a universal wheel distribution and preliminary Django 1.9 support.


This month’s excellence award went to Mythmon’s “Severance Santa” plan for holding layoffs over the holiday break.

If you’re interested in web development at Mozilla, or want to attend next month’s Extravaganza, subscribe to the dev-webdev@lists.mozilla.org mailing list to be notified of the next meeting, and maybe send a message introducing yourself. We’d love to meet you!

See you next month!

Eradicating those nasty .pyc files

I recently acquired a new development laptop and moved a number of local Git repositories from my old machine to my new machine. In doing so I also changed the folder structure, and when trying to run some code I was presented with this Python error:


import file mismatch:
imported module 'tests.desktop.consumer_pages.test_details_page' has this __file__ attribute:
/Users/bsilverberg/gitRepos/marketplace-tests/tests/desktop/consumer_pages/test_details_page.py
which is not the same as the test file we want to collect:
/Users/bsilverberg/Documents/gitRepos/marketplace-tests/tests/desktop/consumer_pages/test_details_page.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules

This was a symptom of the fact that Python creates .pyc files on my machine when it compiles code. This can result in other nastiness too, as well as cluttering up your machine, so I wanted to both delete all of these files and also prevent Python from doing it in the future. This post contains info on how to do both.

Deleting all .pyc files from a folder

You can use the find command (on OS X and Linux) to locate all of the .pyc files, and then use its delete option to delete them.

The command to find all .pyc files in all folders, starting with the current one is:
find . -name '*.pyc'

If you want to delete all the files found, just add the -delete option:
find . -name '*.pyc' -delete

Obviously, this can be used for any file type that you wish to eradicate, not just .pyc files.

Preventing Python from writing .pyc files

I don’t like having all of those extra files cluttering my machine, and, in addition to the error I mentioned above, I have from time to time seen other errors related to out of date .pyc files.

Another issue that .pyc files can cause is that they can be orphaned, for example if you remove a .py file from your project, but the .pyc file remains (which can happen as one often adds *.pyc to .gitignore). Python can then still pick up the module from the .pyc file via an import which can lead to difficult to diagnose bugs.

For these reasons I want to prevent Python from ever writing those files again. To do this all you have to do is set the environment variable PYTHONDONTWRITEBYTECODE to 1. You can ensure that that variable is set for any bash session that you start by adding the following to your .bash_profile or .bashrc:
export PYTHONDONTWRITEBYTECODE=1

Beer and Tell – October 2015

Once a month, web developers from across the Mozilla Project get together to forge Bitcoins. While we magnetize needles for carving out our counterfeit bits directly on hard drive platters, we find time to talk about our side projects and drink, an occurrence we like to call “Beer and Tell”.

There’s a wiki page available with a list of the presenters, as well as links to their presentation materials. There’s also a recording available courtesy of Air Mozilla.

ErikRose: Rubik’s Magic Cube

First up was ErikRose, who talked about the process of learning how to solve a Rubik’s Cube and how it affected his thinking. The learning process mirrored that of learning a language: first, you only see the cube as a block of individual colors, but as you progress you start to recognize specific cubes and arrangements of cubes as “words”, and eventually you become able to recognize abstract patterns of cubes relative to each other.

Check out the recording for a detailed walk through part of the cube-solving process!

Mythmon: N-Bodies Simulation in Rust

Next was Mythmon, who shared a physics simulation of bodies floating in space being affected by gravity. The simulation was written in Rust and relies on Piston for drawing graphics to the screen.

Potch: Canvas Blur

Potch was next with a demo of performing a Gaussian blur on a canvas in JavaScript. Branching off some experiments around auto-cropping algorithms, Potch’s demo processes the blur in chunks and produces results that are comparable to blurs produced in Photoshop, albeit much slower and with less sampling for the blur. While not intended to actually be used for anything, future improvements include using Web Workers to process the blur asynchronously and in parallel, as well as performing other convolutions besides blurring.

BWalker: ASCII Art Dashboard

Last up was bwalker, who demoed an ASCII art dashboard powered by blessed-contrib. The library provides the ability to make console-based dashboards with widgets for line graphs, bar charts, tables, and even a world map. This particular demo pulled some statistics from Github as well as graphing randomly generated numbers.


The highlight of this week’s meetup was lonnen’s impressive feat of creating 5 bitcoins on a flash drive using only a lighter and a chiropractic activator.

If you’re interested in attending the next Beer and Tell, sign up for the dev-webdev@lists.mozilla.org mailing list. An email is sent out a week beforehand with connection details. You could even add yourself to the wiki and show off your side-project!

See you next month!

Extravaganza – October 2015

Once a month, web developers from across Mozilla get together to see who can score the highest car emissions rating. While we scour the world for dangerous combustibles, we find time to talk about the work that we’ve shipped, share the libraries we’re working on, meet new folks, and talk about whatever else is on our minds. It’s the Webdev Extravaganza! The meeting is open to the public; you should stop by!

You can check out the wiki page that we use to organize the meeting, or view a recording of the meeting in Air Mozilla. Or just read on for a summary!

Shipping Celebration

The shipping celebration is for anything we finished and deployed in the past month, whether it be a brand new site, an upgrade to an existing one, or even a release of a library.

One and Done on Heroku

First up was bsilverberg with the news that One and Done, Mozilla’s contributor task board, has successfully migrated from the Mozilla-hosted Stackato PAAS to Heroku. One and Done takes advantage of a few interesting features of Heroku, such as App Pipelines and Review Apps.

Pontoon Sync Improvements and New Leaderboards

Next was Osmose (that’s me!) sharing a few new features on Pontoon, a site for submitting translations for Mozilla software:

  • mathjazz added a new “Latest Activity” column to project and locale listings.
  • jotes added time-based filters to the leaderboard as well as several performance improvements to the page.
  • I added inc file support to the new sync process, as well as a few other fixes such that all Pontoon projects are now using the new sync. Yay!

Air Mozilla iTunes Video Podcast

Peterbe stopped by to share the news that Air Mozilla now has an iTunes-compatible podcast feed for all of its videos. The feed has already been approved by Apple and is available on the iTunes Store.

DXR Static File Cachebusting

ErikRose shared the news that DXR now adds hashes to static file filenames so that updates to the static files don’t get messed up by an old local cache. Interestingly, instead of relying on popular tools like Grunt or Webpack, Erik opted to implement the hashing using some logic in a Makefile plus a little bit of Python. You can check out the pull request to see the details of the change.

Replacing localForage with localStorage

Next we went back to Peterbe, who recently replaced localForage, a library that abstracts several different methods of storing data locally in the browser, with localStorage, the blocking, built-in storage solution that ships with browsers. He also shared a blog post that showed that not only is localStorage simpler to use, it was actually faster in his specific use case.

Mozilla.org Hosted on Deis

Next was jgmize sharing the news that mozilla.org is now hosted on a Deis cluster. Currently a small amount of production data is being hosted by the cluster, but a larger rollout is planned for the near future. Giorgos is responsible for the entire Jenkins setup, including a neat deployment pipeline display, an Ansible playbook, and the deployment pipeline scripts. Pmac ported over hundreds of Apache redirects to Python and wrote a comprehensive set of tests for them as well.

Roundtable

The Roundtable is the home for discussions that don’t fit anywhere else.

Farewell to Wenzel

Lastly, I wanted to specifically call out wenzel, whose last day as a paid contributor for Mozilla was last Friday. Wenzel has been a Mozillian for 9 years, starting as an intern. He’s contributed to almost every major Mozilla web property, and will be missed.


This month’s winner was willkg, with an impressive 200% rating, generating more pollution than the fuel he put in. Local science expert Dr. Potch of Mr. Potch’s Questionable Ethics and Payday Loan Barn was quoted as saying that willkg’s score was “possible”.

If you’re interested in web development at Mozilla, or want to attend next month’s Extravaganza, subscribe to the dev-webdev@lists.mozilla.org mailing list to be notified of the next meeting, and maybe send a message introducing yourself. We’d love to meet you!

See you next month!

Using peep on Heroku

I recently moved a Django app to Heroku which was using peep, rather than pip, for package installation. By default Heroku will use pip to install your required packages when it sees a requirements.txt file in the root of your project, and the option to use peep instead does not exist.

Luckily one of my colleagues, pmac, was kind enough to create a share a Heroku buildpack which uses peep instead of pip for package installation. All you need to do to make use of this buildpack is issue the following command using the Heroku CLI for your app:

heroku buildpacks:set https://github.com/pmclanahan/heroku-buildpack-python-peep

Of course, make sure your requirements.txt is peep-compatible.

Happy peeping!

Beer and Tell – September 2015

Once a month, web developers from across the Mozilla Project get together to design user-friendly advertisements. When we’re not deciding which police alarm we want to auto-play, we find time to talk about our side projects and drink, an occurrence we like to call “Beer and Tell”.

There’s a wiki page available with a list of the presenters, as well as links to their presentation materials. There’s also a recording available courtesy of Air Mozilla.

Erik Rose: Whitespace-significant Meta-grammar for Homoiconic Languages

The only presenter this week was ErikRose, who shared a prototype grammar for a class of whitespace-significant programming languages. The goal was to represent the nested-list structure of homoiconic languages like Lisp while keeping redundant bracing out of the way like Python.

if is_yak then
    shave yak
    bathe yak
  else
    do productive_work

--> (if is_yak then
        ((shave yak)
         (bathe yak))
     else
        ((do productive_work))
    )

Normally, when the else statement in code similar to the sample above isn’t indented and is at the same level as the if statement, the parser must know that else statements may follow after if statements and are related to them. To get around this, ErikRose suggests using a partial outdent. This way, the parser can recognize that the else statement is separate from the shave and bathe lines but should still be grouped with the if statement, even if the parser doesn’t know what an if statement is.

Erik’s gist contains a few more samples and a possible grammar for parsing a language like this using Parsimonious, his parsing library.


Our focus group of QVC customers has given us great insight in to just what the average internet user is looking for in their sponsored content. Early retirement, here we come!

If you’re interested in attending the next Beer and Tell, sign up for the dev-webdev@lists.mozilla.org mailing list. An email is sent out a week beforehand with connection details. You could even add yourself to the wiki and show off your side-project!

See you next month!

Extravaganza – September 2015

Once a month, web developers from across Mozilla get together to to work on our containerization startup. While we buy up the latest stock of Tupperware, we find time to talk about the work that we’ve shipped, share the libraries we’re working on, meet new folks, and talk about whatever else is on our minds. It’s the Webdev Extravaganza! The meeting is open to the public; you should stop by!

You can check out the wiki page that we use to organize the meeting, or view a recording of the meeting in Air Mozilla. Or just read on for a summary!

Shipping Celebration

The shipping celebration is for anything we finished and deployed in the past month, whether it be a brand new site, an upgrade to an existing one, or even a release of a library.

Pontoon

Pontoon is a tool for translating Mozilla software. This past month saw several changes ship, including a brand new sync process that is up to 80% faster than the old sync process. Other changes include a new listing page for locale teams, a 50% speedup on the translation view, several new locales, and several visual improvements.

Crash-Stats Faster Loading

Crash-Stats is a service for analyzing crash reports from various Mozilla products. This month they shipped more aggressive cache headers for static assets, as well as Gzip compression for JavaScript, CSS, and HTML files. The result was a 25% improvement in load time! Further improvements are planned to reduce the time-to-first-byte.

DXR 2.0

DXR is a code search and navigation tool for Mozilla (and other) projects. DXR 2.0 has been in development for a long time and has shipped at last! The new version comes with a host of updates, including:

  • Support for C++ and Rust, as well as limited Python and XPDIL support
  • Parallel, clustered indexing
  • Binary file and image browsing
  • Plugin-based architecture
  • And more!

Open-source Citizenship

Here we talk about libraries we’re maintaining and what, if anything, we need help with for them.

Optisorl

Peterbe shared a library he created called optisorl. It is a pluggable backend for sorl-thumbnail that optimizes thumbnails using pngquant, gifsicle, and, coming soon, mozjpeg for optimizing the generated thumbnails.

Roundtable

The Roundtable is the home for discussions that don’t fit anywhere else.

Hacks Blog Redesign

Potch informed us of his in-progress redesign of the Hacks blog. A demo of his progress is available and he is looking for feedback. Check it out! Tell him how you feel about it!


If you sign up in the next month, you get a free upgrade to Rubbermaid-brand containers! As always, our compute instances are delivered straight to your door and are always fresh, never frozen.

If you’re interested in web development at Mozilla, or want to attend next month’s Extravaganza, subscribe to the dev-webdev@lists.mozilla.org mailing list to be notified of the next meeting, and maybe send a message introducing yourself. We’d love to meet you!

See you next month!

Node.js static file build steps in Python Heroku apps

I write a lot of webapps. I like to use Python for the backend, but most
frontend tools are written in Node.js. LESS gives me nicer style sheets, Babel
lets me write next-generation JavaScript, and NPM helps manage dependencies
nicely. As a result, most of my projects are polyglots that can be difficult to
deploy.

Modern workflows have already figured this out: Run all the tools. Most
READMEs I’ve written lately tend to look like this:

$ git clone https://github.example.com/foo/bar.git
$ cd git
$ pip install -r requirements.txt
$ npm install
$ gulp static-assets
$ python ./manage.py runserver

I like to deploy my projects using Heroku. They take care of the messy details
about deployment, but they don’t seem to support multi-language projects easily.
There are Python and Node buildpacks, but no clear way of combining the two.

Multi Buildpack

GitHub is littered with attempts to fix this by building new buildpacks.
The problem is they invariable fall out of compatibility with Heroku. I could
probably fix, but then I’d have to maintain them. I use Heroku to avoid
maintaining infrastructure; custom buildpacks are one step forward, but two
steps back.

Enter Multi Buildpack, which runs multiple buildpacks at once.

It is simple enough that it is unlike to fall out of compatibility. Heroku has a
fork of the project on their GitHub account, which implies that it will be
maintained in the future.

To configure the buildpack, first tell Heroku you want to use it:

$ heroku buildpacks:set https://github.com/heroku/heroku-buildpack-multi.git

Next, add a .buildpacks file to your project that lists the buildpacks to run:

https://github.com/heroku/heroku-buildpack-nodejs.git
https://github.com/heroku/heroku-buildpack-python.git

Buildpacks are executed in the order they’re listed in, allowing later
buildpacks to use the tools and scripts installed by earlier buildpacks.

The Problem With Python

There’s one problem: The Python buildpack moves files around, which makes it
incompatible with the way the Node buildpack installs commands. This means that
any asset compilation or minification done as a step of the Python buildpack
that depends on Node will fail.

The Python buildpack automatically detects a Django project and runs
./manage.py collectstatic. But the Node environment isn’t available, so this
fails. No static files get built.

There is a solution: bin/post_compile! If present in your repository, this
script will be run at the end of the build process. Because it runs outside of
the Python buildpack, commands installed by the Node buildpack are available and
will work correctly.

This trick works with any Python webapp, but lets use a Django project as an
example. I often use Django Pipeline for static asset compilation. Assets
are compiled using the command ./manage.py collectstatic, which, when properly
configured, will call all the Node commands.

#!/bin/bash
export PATH=/app/.heroku/node/bin:$PATH
./manage.py collectstatic --noinput

Alternatively, you could call Node tools like Gulp or Webpack directly.

In the case of Django Pipeline, it is also useful to disable the Python
buildpack from running collectstatic, since it will fail anyways. This is done
using an environment variable:

heroku config:set DISABLE_COLLECTSTATIC 1

Okay, so there is a little hack here. We still had to append the Node binary
folder to PATH. Pretend you didn’t see that! Or don’t, because you’ll need
to do it in your script too.

That’s it

To recap, this approach:

  1. Only uses buildpacks available from Heroku
  2. Supports any sort of Python and/or Node build steps
  3. Doesn’t require vendoring or pre-compiling any static assets

Woot!