Extravaganza – November 2015

Michael Kelly

Once a month, web developers from across Mozilla get together to revolutionize layoffs. While we calculate the optimum distance away an employee should be from their desk when they’re informed of their impending doom, we find time to talk about the work that we’ve shipped, share the libraries we’re working on, meet new folks, and talk about whatever else is on our minds. It’s the Webdev Extravaganza! The meeting is open to the public; you should stop by!

You can check out the wiki page that we use to organize the meeting, or view a recording of the meeting in Air Mozilla. Or just read on for a summary!

Shipping Celebration

The shipping celebration is for anything we finished and deployed in the past month, whether it be a brand new site, an upgrade to an existing one, or even a release of a library.

Pontoon is replacing Verbatim

Osmose (that’s me!) started us off with the news that Pontoon, a Mozilla-developed website for translating software, is going to replace Verbatim, Mozilla’s instance of Pootle, which is an externally-developed website for translating software. Pontoon is able to do certain things that Pootle doesn’t, such as in-page translation.

Over 60% of active localization teams have voluntarily moved to Pontoon already, and the team currently aims to have 100% of active localization to have moved over by the first week of December.

mozilla.org on Deis Update

Next was jgmize, who shared a small update about the previously-mentioned move of mozilla.org to Deis, an open-source PaaS. The site has now fully transitioned out of the PHX1 datacenter and is split between Deis on AWS and SCL3.

Open-source Citizenship

Here we talk about libraries we’re maintaining and what, if anything, we need help with for them.

django-browserid 1.0.1

Last was Osmose again with the news that django-browserid 1.0.1 has been released. The only changes are the addition of a universal wheel distribution and preliminary Django 1.9 support.

This month’s excellence award went to Mythmon’s “Severance Santa” plan for holding layoffs over the holiday break.

If you’re interested in web development at Mozilla, or want to attend next month’s Extravaganza, subscribe to the dev-webdev@lists.mozilla.org mailing list to be notified of the next meeting, and maybe send a message introducing yourself. We’d love to meet you!

See you next month!

Eradicating those nasty .pyc files

Bob Silverberg

I recently acquired a new development laptop and moved a number of local Git repositories from my old machine to my new machine. In doing so I also changed the folder structure, and when trying to run some code I was presented with this Python error:


import file mismatch:
imported module 'tests.desktop.consumer_pages.test_details_page' has this __file__ attribute:
/Users/bsilverberg/gitRepos/marketplace-tests/tests/desktop/consumer_pages/test_details_page.py
which is not the same as the test file we want to collect:
/Users/bsilverberg/Documents/gitRepos/marketplace-tests/tests/desktop/consumer_pages/test_details_page.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules

This was a symptom of the fact that Python creates .pyc files on my machine when it compiles code. This can result in other nastiness too, as well as cluttering up your machine, so I wanted to both delete all of these files and also prevent Python from doing it in the future. This post contains info on how to do both.

Deleting all .pyc files from a folder

You can use the find command (on OS X and Linux) to locate all of the .pyc files, and then use its delete option to delete them.

The command to find all .pyc files in all folders, starting with the current one is:
find . -name '*.pyc'

If you want to delete all the files found, just add the -delete option:
find . -name '*.pyc' -delete

Obviously, this can be used for any file type that you wish to eradicate, not just .pyc files.

Preventing Python from writing .pyc files

I don’t like having all of those extra files cluttering my machine, and, in addition to the error I mentioned above, I have from time to time seen other errors related to out of date .pyc files.

Another issue that .pyc files can cause is that they can be orphaned, for example if you remove a .py file from your project, but the .pyc file remains (which can happen as one often adds *.pyc to .gitignore). Python can then still pick up the module from the .pyc file via an import which can lead to difficult to diagnose bugs.

For these reasons I want to prevent Python from ever writing those files again. To do this all you have to do is set the environment variable PYTHONDONTWRITEBYTECODE to 1. You can ensure that that variable is set for any bash session that you start by adding the following to your .bash_profile or .bashrc:
export PYTHONDONTWRITEBYTECODE=1

Beer and Tell – October 2015

Michael Kelly

Once a month, web developers from across the Mozilla Project get together to forge Bitcoins. While we magnetize needles for carving out our counterfeit bits directly on hard drive platters, we find time to talk about our side projects and drink, an occurrence we like to call “Beer and Tell”.

There’s a wiki page available with a list of the presenters, as well as links to their presentation materials. There’s also a recording available courtesy of Air Mozilla.

ErikRose: Rubik’s Magic Cube

First up was ErikRose, who talked about the process of learning how to solve a Rubik’s Cube and how it affected his thinking. The learning process mirrored that of learning a language: first, you only see the cube as a block of individual colors, but as you progress you start to recognize specific cubes and arrangements of cubes as “words”, and eventually you become able to recognize abstract patterns of cubes relative to each other.

Check out the recording for a detailed walk through part of the cube-solving process!

Mythmon: N-Bodies Simulation in Rust

Next was Mythmon, who shared a physics simulation of bodies floating in space being affected by gravity. The simulation was written in Rust and relies on Piston for drawing graphics to the screen.

Potch: Canvas Blur

Potch was next with a demo of performing a Gaussian blur on a canvas in JavaScript. Branching off some experiments around auto-cropping algorithms, Potch’s demo processes the blur in chunks and produces results that are comparable to blurs produced in Photoshop, albeit much slower and with less sampling for the blur. While not intended to actually be used for anything, future improvements include using Web Workers to process the blur asynchronously and in parallel, as well as performing other convolutions besides blurring.

BWalker: ASCII Art Dashboard

Last up was bwalker, who demoed an ASCII art dashboard powered by blessed-contrib. The library provides the ability to make console-based dashboards with widgets for line graphs, bar charts, tables, and even a world map. This particular demo pulled some statistics from Github as well as graphing randomly generated numbers.

The highlight of this week’s meetup was lonnen’s impressive feat of creating 5 bitcoins on a flash drive using only a lighter and a chiropractic activator.

If you’re interested in attending the next Beer and Tell, sign up for the dev-webdev@lists.mozilla.org mailing list. An email is sent out a week beforehand with connection details. You could even add yourself to the wiki and show off your side-project!

See you next month!

Extravaganza – October 2015

Michael Kelly

Once a month, web developers from across Mozilla get together to see who can score the highest car emissions rating. While we scour the world for dangerous combustibles, we find time to talk about the work that we’ve shipped, share the libraries we’re working on, meet new folks, and talk about whatever else is on our minds. It’s the Webdev Extravaganza! The meeting is open to the public; you should stop by!

You can check out the wiki page that we use to organize the meeting, or view a recording of the meeting in Air Mozilla. Or just read on for a summary!

Shipping Celebration

The shipping celebration is for anything we finished and deployed in the past month, whether it be a brand new site, an upgrade to an existing one, or even a release of a library.

One and Done on Heroku

First up was bsilverberg with the news that One and Done, Mozilla’s contributor task board, has successfully migrated from the Mozilla-hosted Stackato PAAS to Heroku. One and Done takes advantage of a few interesting features of Heroku, such as App Pipelines and Review Apps.

Pontoon Sync Improvements and New Leaderboards

Next was Osmose (that’s me!) sharing a few new features on Pontoon, a site for submitting translations for Mozilla software:

  • mathjazz added a new “Latest Activity” column to project and locale listings.
  • jotes added time-based filters to the leaderboard as well as several performance improvements to the page.
  • I added inc file support to the new sync process, as well as a few other fixes such that all Pontoon projects are now using the new sync. Yay!

Air Mozilla iTunes Video Podcast

Peterbe stopped by to share the news that Air Mozilla now has an iTunes-compatible podcast feed for all of its videos. The feed has already been approved by Apple and is available on the iTunes Store.

DXR Static File Cachebusting

ErikRose shared the news that DXR now adds hashes to static file filenames so that updates to the static files don’t get messed up by an old local cache. Interestingly, instead of relying on popular tools like Grunt or Webpack, Erik opted to implement the hashing using some logic in a Makefile plus a little bit of Python. You can check out the pull request to see the details of the change.

Replacing localForage with localStorage

Next we went back to Peterbe, who recently replaced localForage, a library that abstracts several different methods of storing data locally in the browser, with localStorage, the blocking, built-in storage solution that ships with browsers. He also shared a blog post that showed that not only is localStorage simpler to use, it was actually faster in his specific use case.

Mozilla.org Hosted on Deis

Next was jgmize sharing the news that mozilla.org is now hosted on a Deis cluster. Currently a small amount of production data is being hosted by the cluster, but a larger rollout is planned for the near future. Giorgos is responsible for the entire Jenkins setup, including a neat deployment pipeline display, an Ansible playbook, and the deployment pipeline scripts. Pmac ported over hundreds of Apache redirects to Python and wrote a comprehensive set of tests for them as well.

Roundtable

The Roundtable is the home for discussions that don’t fit anywhere else.

Farewell to Wenzel

Lastly, I wanted to specifically call out wenzel, whose last day as a paid contributor for Mozilla was last Friday. Wenzel has been a Mozillian for 9 years, starting as an intern. He’s contributed to almost every major Mozilla web property, and will be missed.

This month’s winner was willkg, with an impressive 200% rating, generating more pollution than the fuel he put in. Local science expert Dr. Potch of Mr. Potch’s Questionable Ethics and Payday Loan Barn was quoted as saying that willkg’s score was “possible”.

If you’re interested in web development at Mozilla, or want to attend next month’s Extravaganza, subscribe to the dev-webdev@lists.mozilla.org mailing list to be notified of the next meeting, and maybe send a message introducing yourself. We’d love to meet you!

See you next month!

Using peep on Heroku

Bob Silverberg

I recently moved a Django app to Heroku which was using peep, rather than pip, for package installation. By default Heroku will use pip to install your required packages when it sees a requirements.txt file in the root of your project, and the option to use peep instead does not exist.

Luckily one of my colleagues, pmac, was kind enough to create a share a Heroku buildpack which uses peep instead of pip for package installation. All you need to do to make use of this buildpack is issue the following command using the Heroku CLI for your app:

heroku buildpacks:set https://github.com/pmclanahan/heroku-buildpack-python-peep

Of course, make sure your requirements.txt is peep-compatible.

Happy peeping!

Beer and Tell – September 2015

Michael Kelly

Once a month, web developers from across the Mozilla Project get together to design user-friendly advertisements. When we’re not deciding which police alarm we want to auto-play, we find time to talk about our side projects and drink, an occurrence we like to call “Beer and Tell”.

There’s a wiki page available with a list of the presenters, as well as links to their presentation materials. There’s also a recording available courtesy of Air Mozilla.

Erik Rose: Whitespace-significant Meta-grammar for Homoiconic Languages

The only presenter this week was ErikRose, who shared a prototype grammar for a class of whitespace-significant programming languages. The goal was to represent the nested-list structure of homoiconic languages like Lisp while keeping redundant bracing out of the way like Python.

if is_yak then
    shave yak
    bathe yak
  else
    do productive_work

--> (if is_yak then
        ((shave yak)
         (bathe yak))
     else
        ((do productive_work))
    )

Normally, when the else statement in code similar to the sample above isn’t indented and is at the same level as the if statement, the parser must know that else statements may follow after if statements and are related to them. To get around this, ErikRose suggests using a partial outdent. This way, the parser can recognize that the else statement is separate from the shave and bathe lines but should still be grouped with the if statement, even if the parser doesn’t know what an if statement is.

Erik’s gist contains a few more samples and a possible grammar for parsing a language like this using Parsimonious, his parsing library.

Our focus group of QVC customers has given us great insight in to just what the average internet user is looking for in their sponsored content. Early retirement, here we come!

If you’re interested in attending the next Beer and Tell, sign up for the dev-webdev@lists.mozilla.org mailing list. An email is sent out a week beforehand with connection details. You could even add yourself to the wiki and show off your side-project!

See you next month!

Extravaganza – September 2015

Michael Kelly

Once a month, web developers from across Mozilla get together to to work on our containerization startup. While we buy up the latest stock of Tupperware, we find time to talk about the work that we’ve shipped, share the libraries we’re working on, meet new folks, and talk about whatever else is on our minds. It’s the Webdev Extravaganza! The meeting is open to the public; you should stop by!

You can check out the wiki page that we use to organize the meeting, or view a recording of the meeting in Air Mozilla. Or just read on for a summary!

Shipping Celebration

The shipping celebration is for anything we finished and deployed in the past month, whether it be a brand new site, an upgrade to an existing one, or even a release of a library.

Pontoon

Pontoon is a tool for translating Mozilla software. This past month saw several changes ship, including a brand new sync process that is up to 80% faster than the old sync process. Other changes include a new listing page for locale teams, a 50% speedup on the translation view, several new locales, and several visual improvements.

Crash-Stats Faster Loading

Crash-Stats is a service for analyzing crash reports from various Mozilla products. This month they shipped more aggressive cache headers for static assets, as well as Gzip compression for JavaScript, CSS, and HTML files. The result was a 25% improvement in load time! Further improvements are planned to reduce the time-to-first-byte.

DXR 2.0

DXR is a code search and navigation tool for Mozilla (and other) projects. DXR 2.0 has been in development for a long time and has shipped at last! The new version comes with a host of updates, including:

  • Support for C++ and Rust, as well as limited Python and XPDIL support
  • Parallel, clustered indexing
  • Binary file and image browsing
  • Plugin-based architecture
  • And more!

Open-source Citizenship

Here we talk about libraries we’re maintaining and what, if anything, we need help with for them.

Optisorl

Peterbe shared a library he created called optisorl. It is a pluggable backend for sorl-thumbnail that optimizes thumbnails using pngquant, gifsicle, and, coming soon, mozjpeg for optimizing the generated thumbnails.

Roundtable

The Roundtable is the home for discussions that don’t fit anywhere else.

Hacks Blog Redesign

Potch informed us of his in-progress redesign of the Hacks blog. A demo of his progress is available and he is looking for feedback. Check it out! Tell him how you feel about it!

If you sign up in the next month, you get a free upgrade to Rubbermaid-brand containers! As always, our compute instances are delivered straight to your door and are always fresh, never frozen.

If you’re interested in web development at Mozilla, or want to attend next month’s Extravaganza, subscribe to the dev-webdev@lists.mozilla.org mailing list to be notified of the next meeting, and maybe send a message introducing yourself. We’d love to meet you!

See you next month!

Node.js static file build steps in Python Heroku apps

Mike Cooper

1 response

I write a lot of webapps. I like to use Python for the backend, but most
frontend tools are written in Node.js. LESS gives me nicer style sheets, Babel
lets me write next-generation JavaScript, and NPM helps manage dependencies
nicely. As a result, most of my projects are polyglots that can be difficult to
deploy.

Modern workflows have already figured this out: Run all the tools. Most
READMEs I’ve written lately tend to look like this:

$ git clone https://github.example.com/foo/bar.git
$ cd git
$ pip install -r requirements.txt
$ npm install
$ gulp static-assets
$ python ./manage.py runserver

I like to deploy my projects using Heroku. They take care of the messy details
about deployment, but they don’t seem to support multi-language projects easily.
There are Python and Node buildpacks, but no clear way of combining the two.

Multi Buildpack

GitHub is littered with attempts to fix this by building new buildpacks.
The problem is they invariable fall out of compatibility with Heroku. I could
probably fix, but then I’d have to maintain them. I use Heroku to avoid
maintaining infrastructure; custom buildpacks are one step forward, but two
steps back.

Enter Multi Buildpack, which runs multiple buildpacks at once.

It is simple enough that it is unlike to fall out of compatibility. Heroku has a
fork of the project on their GitHub account, which implies that it will be
maintained in the future.

To configure the buildpack, first tell Heroku you want to use it:

$ heroku buildpacks:set https://github.com/heroku/heroku-buildpack-multi.git

Next, add a .buildpacks file to your project that lists the buildpacks to run:

https://github.com/heroku/heroku-buildpack-nodejs.git
https://github.com/heroku/heroku-buildpack-python.git

Buildpacks are executed in the order they’re listed in, allowing later
buildpacks to use the tools and scripts installed by earlier buildpacks.

The Problem With Python

There’s one problem: The Python buildpack moves files around, which makes it
incompatible with the way the Node buildpack installs commands. This means that
any asset compilation or minification done as a step of the Python buildpack
that depends on Node will fail.

The Python buildpack automatically detects a Django project and runs
./manage.py collectstatic. But the Node environment isn’t available, so this
fails. No static files get built.

There is a solution: bin/post_compile! If present in your repository, this
script will be run at the end of the build process. Because it runs outside of
the Python buildpack, commands installed by the Node buildpack are available and
will work correctly.

This trick works with any Python webapp, but lets use a Django project as an
example. I often use Django Pipeline for static asset compilation. Assets
are compiled using the command ./manage.py collectstatic, which, when properly
configured, will call all the Node commands.

#!/bin/bash
export PATH=/app/.heroku/node/bin:$PATH
./manage.py collectstatic --noinput

Alternatively, you could call Node tools like Gulp or Webpack directly.

In the case of Django Pipeline, it is also useful to disable the Python
buildpack from running collectstatic, since it will fail anyways. This is done
using an environment variable:

heroku config:set DISABLE_COLLECTSTATIC 1

Okay, so there is a little hack here. We still had to append the Node binary
folder to PATH. Pretend you didn’t see that! Or don’t, because you’ll need
to do it in your script too.

That’s it

To recap, this approach:

  1. Only uses buildpacks available from Heroku
  2. Supports any sort of Python and/or Node build steps
  3. Doesn’t require vendoring or pre-compiling any static assets

Woot!

Beer and Tell – August 2015

Michael Kelly

Once a month, web developers from across the Mozilla Project get together to spend an hour of overtime to honor our white-collar brethren at Amazon. As we watch our productivity fall, we find time to talk about our side projects and drink, an occurrence we like to call “Beer and Tell”.

There’s a wiki page available with a list of the presenters, as well as links to their presentation materials. There’s also a recording available courtesy of Air Mozilla.

openjck: Discord

openjck was up first and shared Discord, a Github webhook that scans pull requests for CSS compatibility issues. When it finds an issue, it leaves a comment on the offending line with a short description and which browsers are affected. The check is powered by doiuse, and projects can add a .doiuse file (using browserslist syntax) that specifies which browser versions they want to be tested against. Discord currently checks CSS and Stylus files.

The Discord team (mrmakeit, groovecoder, davidwalsh, and openjck) is looking for sites to test Discord out. Work on the site is currently suspended (which is why it’s a side project, the team may work on it in their free time) so that feedback can be gathered to determine where the site should go next. If you’re interested in trying out Discord, let groovecoder know!

peterbe: Activity and Fanout.io

Next up was peterbe, with an update to Activity. The site now uses Fanout.io and a message queue to improve how activity items are fetched from GitHub and other sources. The site queues up jobs to fetch data from the Github API, and as the jobs complete, they send their results to Fanout. Fanout’s JavaScript library maintains an open WebSocket with their service, and when Fanout receives the data from the completed jobs, it notifies the client of the new data, which gets written to localStorage and updates the React state. This allows Activity to remain structured as an offline-ready application while still receiving seamless updates if the user has an internet connection.

There’s a donation jar near the exit; for just $25 you can pay for an hour of time for an Amazon engineer to spend with their family. Checks may be made payable to No Questions Asked Laundry, LLC.

If you’re interested in attending the next Beer and Tell, sign up for the dev-webdev@lists.mozilla.org mailing list. An email is sent out a week beforehand with connection details. You could even add yourself to the wiki and show off your side-project!

See you next month!

Extravaganza – August 2015

Michael Kelly

1 response

Once a month, web developers from across Mozilla get together to recreate classic dishes using only soylent and food coloring. As we discover what oven temperatures are safe for the techno-wizard goop, we find time to talk about the work that we’ve shipped, share the libraries we’re working on, meet new folks, and talk about whatever else is on our minds. It’s the Webdev Extravaganza! The meeting is open to the public; you should stop by!

You can check out the wiki page that we use to organize the meeting, or view a recording of the meeting in Air Mozilla. Or just read on for a summary!

Shipping Celebration

The shipping celebration is for anything we finished and deployed in the past month, whether it be a brand new site, an upgrade to an existing one, or even a release of a library.

Pontoon: One L10n Platform to Rule Them All

First up was Osmose (that’s me!) sharing the news that Pontoon is now able to localize Firefox, Thunderbird, and other Gecko-based apps, making it a single tool capable of localizing every project across Mozilla! The Slovenian localization team is already using it for all the projects they actively translate, and more locales are signing up every day.

New Hires / Interns / Volunteers / Contributors

Here we introduce any newcomers to the Webdev group, including new employees, interns, volunteers, or any other form of contributor.

Farewell to new_one and nolski!

Intern season is winding down, and that means beloved Webdev interns new_one and nolski are leaving us! We wish them well and thank them for their hard work. New_one already gave a presentation on his contributions to DXR, and nolski’s presentation will be streamed on Air Mozilla on August 11th.

Roundtable

The Roundtable is the home for discussions that don’t fit anywhere else.

React Devtools in Firefox

Mythmon shared the news that the React Developer Tools extension has released a pre-release Firefox add-on. The extension allows developers working on React-based webapps to inspect the component hierarchy on their page as well as modify state and props for debugging purposes.

While we haven’t really nailed that “taste” thing, we were able to make imitation caesar dressing by pouring the soylent into an old Hidden Valley bottle!

If you’re interested in web development at Mozilla, or want to attend next month’s Extravaganza, subscribe to the dev-webdev@lists.mozilla.org mailing list to be notified of the next meeting, and maybe send a message introducing yourself. We’d love to meet you!

See you next month!