Django, Pipeline, and Gulp

Bedrock, the code behind www.mozilla.org, is a very large Django project. It is mostly large due to the volume and diversity of independent pages it serves. These pages come with a surprising amount of static media (css, js, images, fonts, etc.). So, any system that we use to deal with said media should be efficient in order to keep our development servers fast.

We like django-pipeline for managing our static media in production. It does a great job of bundling, minifying, and compressing our css and js. When using it in a development environment however, it does not scale well. The issue is that it does not watch for changes to your files, so all it can do is copy them all from their source to the static directory on every page load. For a reasonable number of files this is probably fine, but as I said ours is not that. This is exasperated in slow I/O environments like Docker on a non-linux system (like OSX). We’ve not been able to setup an acceptable Docker-based local dev environment yet because it can literally take several minutes to render the home page.

Due to all of the issues noted above we’ve been looking for other ways of handling static media. We’ve considered a few times to move to a completely nodejs based system that would be completely independent of the Django side, and may still do that some day, but the problem has always been scope and impact. Again because the project is so large, making sweeping changes that affect workflow and all static files can both take a lot of time and be very disruptive when they land. So for a long time we figured we were stuck. But recently a conversation started in IRC about being able to just disable django-pipeline’s copying of files. If we could do that we could use gulp-watch to much more quickly and efficiently manage these files while being edited and still get the benefits of django-pipline for production. It turned out that someone else already had this idea and mostly we just needed to upgrade django-pipeline.

After that it was a simple matter of adding a task to our Gulpfile:

gulp.task('media:watch', function () {
    return gulp.src('./media/**/*')
        .pipe(watch('./media/**/*', {
            'verbose': true
        }))
        .pipe(gulp.dest('./static'));
});

But it was still a bit odd now having to have two shells open, one for the gulp task and another for the Django dev server. So we did a little more gulp magic and now have a single command to start up both the file watching and the Django server that combines the output of both in a single terminal.

gulp.task('serve:backend', function () {
    var devServerPort = process.env.PORT || 8000;
    process.env.PYTHONUNBUFFERED = 1;
    process.env.PYTHONDONTWRITEBITECODE = 1;
    spawn('python', ['manage.py', 'runserver', '0.0.0.0:' + devServerPort], {
        stdio: 'inherit'
    });
});

gulp.task('default', function() {
    gulp.start('serve:backend');
    gulp.start('media:watch');
});

You can see the full gulpfile.js in our repo on Github if you’d like. It’s a simple but very effective change and has made a very noticeable improvement in the speed of the local development server. And now we hope that we can finally complete and recommend Docker as the default setup for local development on bedrock.

Extravaganza – May 2016

Once a month, web developers from across Mozilla get together to talk about the work that we’ve shipped, share the libraries we’re working on, meet new folks, and talk about whatever else is on our minds. It’s the Webdev Extravaganza! The meeting is open to the public; you should stop by!

You can check out the wiki page that we use to organize the meeting, or view a recording of the meeting in Air Mozilla. Or just read on for a summary!

Shipping Celebration

The shipping celebration is for anything we finished and deployed in the past month, whether it be a brand new site, an upgrade to an existing one, or even a release of a library.

Normandy, the Recipe Server

First up was Osmose (that’s me!), sharing the news that Normandy has shipped! Normandy is a service that will eventually power several Firefox features that involve interacting with users and testing changes to Firefox quickly and safely, such as recommending features that may be useful to users or offering opportunities to try out changes. Right now the service is powering Heartbeat surveys being sent to release users.

Big thanks to the User Advocacy and Web Engineering teams for working on the project!

MDN Save Draft Feature

Next was shobson who talked about MDN‘s Safe Draft feature. When editing an MDN article, the site autosaves your edits to localStorage (if it’s available). Then, when you revisit the editing interface later, the site offers to let you restore or discard the draft, disabling autosave until a decision is made. Future improvements may include previewing drafts and notifying users when an article has changed since their draft was saved.

Air Mozilla Thumbnails

peterbe stopped by to talk about Air Mozilla‘s chapters feature, which allows users to mark and link to segments in a video. The site now auto-generates thumbnails for chapters to help preview what the chapter is about.

Roundtable

The Roundtable is the home for discussions that don’t fit anywhere else.

Docker Development Environments

Last up was jgmize, who asked about use of Docker for easy development environments. The general consensus was that most of the developers present had tried using Dockerized development environments, but tended towards using it only for deployed services or not at all.

Some of the interesting projects brought up for using Docker for development or deployment were:

Check ’em out!


If you’re interested in web development at Mozilla, or want to attend next month’s Extravaganza, subscribe to the dev-webdev@lists.mozilla.org mailing list to be notified of the next meeting, and maybe send a message introducing yourself. We’d love to meet you!

See you next month!

Beer and Tell – April 2016

Once a month, web developers from across the Mozilla Project get together to talk about our side projects and drink, an occurrence we like to call “Beer and Tell”.

There’s a wiki page available with a list of the presenters, as well as links to their presentation materials. There’s also a recording available courtesy of Air Mozilla.

emceeaich: Memory Landscapes

First up was emceeaich, who shared Memory Landscapes, a visual memoir of the life and career of artist and photographer Laurie Toby Edison. The project is presented as a non-linear collection of photographs, in contrast to the traditionally linear format of memoirs. The feature that emceeaich demoed was “Going to Brooklyn”, which gives any link a 1/5 chance of showing shadow pictures briefly before moving on to the linked photo.

lorchard: DIY Keyboard Prototype

Next was lorchard, who talked about the process of making a DIY keyboard using web-based tools. He used keyboard-layout-editor.com to generate a layout serialized in JSON, and then used Plate & Case Builder to generate a CAD file for use with a laser cutter.

A flickr album is available with photos of the process.

lorchard: Jupyter Notebooks in Space

lorchard also shared eve-market-fun, a Node.js-based service that pulls data from the EVE Online API and pre-digests useful information about it. He then uses a Jupyter notebook to pull data from the API and analyze it to guide his market activities in the game. Neat!

Pomax: React Circle-Tree Visualizer

Pomax was up next with a new React component: react-circletree! It depicts a tree structure using segmented concentric circles. The component is very configurable and can by styled with CSS as it is generated via SVG. While built as a side-project, the component can be seen in use on the Web Literacy Framework website.

Pomax: HTML5 Mahjong

Also presented by Pomax was an HTML5 multiplayer Mahjong game. It allows four players to play the classic Chinese game online by using socket.io and a Node.js server to connect the players. The frontend is built using React and Webpack.

groovecoder and John Dungan: Codesy

Last up was groovecoder and John Dungan, who shared codesy, an open-source startup addressing the problem of compensation for fixing bugs in open-source software. They provide a browser extension that allows users to bid on bugs as well as name their price for fixing a bug. Users may then provide proof that they fixed a bug, and once it is approved by the bidders, they receive a payout.


If you’re interested in attending the next Beer and Tell, sign up for the dev-webdev@lists.mozilla.org mailing list. An email is sent out a week beforehand with connection details. You could even add yourself to the wiki and show off your side-project!

See you next month!

Extravaganza – April 2016

Once a month, web developers from across Mozilla get together to talk about the work that we’ve shipped, share the libraries we’re working on, meet new folks, and talk about whatever else is on our minds. It’s the Webdev Extravaganza! The meeting is open to the public; you should stop by!

You can check out the wiki page that we use to organize the meeting, or view a recording of the meeting in Air Mozilla. Or just read on for a summary!

Shipping Celebration

The shipping celebration is for anything we finished and deployed in the past month, whether it be a brand new site, an upgrade to an existing one, or even a release of a library.

A-Frame v0.2.0

First up was ngoke, who shared the news that A-Frame shipped a new release! The v0.2.0 release focuses on extensibility by improving the Component API and adding a slew of new objects that can be registered for use in the framework. A blog post is available that explains the changes.

Let’s Encrypt Switches to Pip 8

ErikRose wasn’t able to attend, but left notes on the wiki about Let’s Encrypt switching from peep to pip 8 for installing its requirements.

DXR Indexing Most Mozilla-Central Branches

Erik also left a note mentioning that DXR, Mozilla’s code browser and search tool for the Firefox codebase, now indexes more branches from mozilla-central, including Aurora, Beta, Central, ESR45, and Release.

Pipstrap 1.1.1

The last note that Erik left mentioned that Pipstrap, a small script for bootstrapping a hash-checked version of Pip, shipped a new release that fixes an error reporting bug under Python 2.6.

Snippets in AWS + Deis

Next up was bensternthal, who shared the news that snippets.mozilla.com is now hosted on Engagement Engineering’s Deis cluster in AWS. Thanks to giorgos, jgmize, and pmac for helping to move the site over! Ben also mentioned that Basket is the next service slated to move over.

Open-source Citizenship

Here we talk about libraries we’re maintaining and what, if anything, we need help with for them.

New django-csp Maintainer

pmac stopped by to let us know that he is taking over maintenance of django-csp. Thanks to jsocol for authoring and maintaining the library until now!

Hashin Python Version Filtering

Last up was peterbe, who shared news of a new release of hashin. The main addition is a --python-version flag, which allows you to filter the hashes hashin adds by python versions that you plan to use, reducing the number hashes added. Thanks to mythmon for submitting the patch!


If you’re interested in web development at Mozilla, or want to attend next month’s Extravaganza, subscribe to the dev-webdev@lists.mozilla.org mailing list to be notified of the next meeting, and maybe send a message introducing yourself. We’d love to meet you!

See you next month!

Beer and Tell – March 2016

Once a month, web developers from across the Mozilla Project get together to talk about our side projects and drink, an occurrence we like to call “Beer and Tell”.

There’s a wiki page available with a list of the presenters, as well as links to their presentation materials. There’s also a recording available courtesy of Air Mozilla.

Osmose: Mandelbrot in Rust

First up was Osmose, who showed off a Rust program he “wrote” that generates a PNG of the Mandelbrot set. And by “wrote”, he means he copied it from an in-progress book on Rust called “Programming Rust” by fellow Mozillian jimb. The book uses the Mandelbrot example to illustrate how easy writing safe parallel code in Rust is. The book is available as an in-progress Early Release from O’Reilly, and is aimed towards people with prior programming experience, especially in another systems programming language.

Peterbe: PodcastTime

Next was peterbe, show shared PodcastTime, a directory of over 13 years of podcast metadata that shows you how much time per day, week, and month you need to spend listening to your podcasts to keep up with them. The site also shows a calendar of upcoming and past podcasts in your list, and lets you share your list of podcasts with others.


If you’re interested in attending the next Beer and Tell, sign up for the dev-webdev@lists.mozilla.org mailing list. An email is sent out a week beforehand with connection details. You could even add yourself to the wiki and show off your side-project!

See you next month!

Extravaganza – March 2016

Once a month, web developers from across Mozilla get together to talk about the work that we’ve shipped, share the libraries we’re working on, meet new folks, and talk about whatever else is on our minds. It’s the Webdev Extravaganza! The meeting is open to the public; you should stop by!

You can check out the wiki page that we use to organize the meeting, or view a recording of the meeting in Air Mozilla. Or just read on for a summary!

Shipping Celebration

The shipping celebration is for anything we finished and deployed in the past month, whether it be a brand new site, an upgrade to an existing one, or even a release of a library.

Peep 3.1.1

First up was ErikRose with news of a new release of Peep. The new version fixes the peep port command so that it correctly emits URL-based requirements.

Erik also wants to remind everyone to drop Peep and upgrade to Pip 8.

SHIELD Recipe Server

Next up was mythmon, who talked about the Recipe Server, a system to help Firefox respond to user issues and test new features quickly and easily. The development server for the service is now live, and is deployed to AWS using Docker and gunicorn.

Go Faster System Addon Update

Next we heard from laura who shared the news that the first Go Faster system addon update shipped! System addons are addons that come bundled with Firefox, but are updated outside of the normal 6-8 week release process. They enable us to try new features out and respond to user feedback quicker than before.

Roundtable

The Roundtable is the home for discussions that don’t fit anywhere else.

Debugging Python with gdb

ErikRose next talked about how he dealt with a segfault in DXR‘s indexing job. The end result of his trial is a description of how to debug Python stack frames using gdb on Ubuntu Trusty.

Conditional Requirements in Python are Bad

Erik also shared a story about conditional requirements in Python. Because Python uses executable Python files for describing the requirements for packages, some projects change their requirements list based on the environment. For example, some projects detect whether they’re running in Python 2.6 or 2.7 and include libraries that implement features that may be missing in Python 2.6.

The problem is that pip caches wheels after evaluating setup.py files, meaning that a cached package built for Python 2.6 may be used when installing in Python 2.7. The solution is to use a feature called environment markers that allows packages to specify requirements against the execution environment that they’re installed to. That way package tooling can check these conditions and avoid using cached packages that aren’t appropriate for the current environment.


If you’re interested in web development at Mozilla, or want to attend next month’s Extravaganza, subscribe to the dev-webdev@lists.mozilla.org mailing list to be notified of the next meeting, and maybe send a message introducing yourself. We’d love to meet you!

See you next month!

Beer and Tell – February 2016

Once a month, web developers from across the Mozilla Project get together to talk about our side projects and drink, an occurrence we like to call "Beer and Tell".

There's a wiki page available with a list of the presenters, as well as links to their presentation materials. There's also a recording available courtesy of Air Mozilla.

Bruce Banner: Web Developer

shobson was up first with Bruce Banner: Web Developer, a small webcomic generator. It provides sweet relief from those workplace stressors via the violent justice of The Incredible Hulk. The idea came from willkg, the code from shobson, and the art from craigcook. Excelsior!

Dokku + Let's Encrypt

Next up was pmac, who showed off dokku, a small PaaS implementation similar to Heroku, that uses Docker containers. Not only is it convenient for running several apps on a single server, but there is also a plugin called dokku-letsencrypt that lets you automatically retrieve and install TLS certificates from letsencrypt.org. Easy peasy!

RPG Maker MV

Next was Osmose (that's me!) who talked about RPG Maker MV, the latest entry in the RPG Maker series of game-making tools. Interestingly, RPGMV uses HTML and JavaScript to implement the engine used to run games made with it. The application itself edits JSON files that are loaded by the web-based engine. The engine itself uses pixi.js for rendering, and can be extended via plugins written in JavaScript.

Battleshits

peterbe stopped by to share Battleshits, a mobile-friendly web app and a fairly gross version of the popular boardgame Battleship. The game connects you with other players via WebSockets and Fanout, and most of the interface is implemented using React.

Chava

Last up was jpetto with a small personal project memorializing his local coffeeshop, Chava, which closed earlier this year. The page uses Hammer.js for touch events and LazyLoad to lazily load the images, but the lightbox implementation is custom-made from scratch. Neato!


If you're interested in attending the next Beer and Tell, sign up for the dev-webdev@lists.mozilla.org mailing list. An email is sent out a week beforehand with connection details. You could even add yourself to the wiki and show off your side-project!

See you next month!

Extravaganza – February 2016

Once a month, web developers from across Mozilla get together to talk about the work that we’ve shipped, share the libraries we’re working on, meet new folks, and talk about whatever else is on our minds. It’s the Webdev Extravaganza! The meeting is open to the public; you should stop by!

You can check out the wiki page that we use to organize the meeting, or view a recording of the meeting in Air Mozilla. Or just read on for a summary!

Shipping Celebration

The shipping celebration is for anything we finished and deployed in the past month, whether it be a brand new site, an upgrade to an existing one, or even a release of a library.

Git Submodules are Gone from MDN

First up was jezdez with news about MDN moving away from using git submodules to pull in dependencies. Instead, MDN now uses pip to pull in dependencies during deployment. Hooray!

Careers now on AWS/Deis

Next was giorgos who let us know that careers.mozilla.org has moved over to the Engagement Engineering Deis cluster on AWS. For deployment, the site has Travis CI build a Docker image and run tests against it. If the tests pass, the image is deployed directly to Deis. Neat!

Privacy Day

jpetto helped ship the Privacy Day page. It includes a mailing list signup form as well as instructions for several platforms on how to update your software to stay secure.

Automated Functional Testing for Mozilla.org

agibson shared news about the migration of previously-external functional tests for mozilla.org to live within the Bedrock repository itself. This allows us to run the tests, which previously were run by the WebQA team against live environments, whenever the site is deployed to dev, stage, or production. Having the functional tests be a part of the build pipeline ensures that developers are aware when the tests are broken and can fix them before deploying broken features. A slide deck is available with more details.

Peep 3.x

ErikRose shared news about the 3.0 (and 3.1) release of Peep, which helps smooth the transition from Peep to Pip 8, which now supports hashed requirements natively. The new Peep includes a peep port command for porting Peep-compatible requirements files to the new Pip 8 format.

Open-source Citizenship

Here we talk about libraries we’re maintaining and what, if anything, we need help with for them.

Jazzband

jezdez shared news about JazzBand, a cooperative experiment to reduce the stress of maintaining Open Source software alone. The group operates as a Github organization that anyone can join and transfer projects to. Anyone in the JazzBand can access JazzBand projects, allowing projects that would otherwise die due to lack of activity thrive thanks to the community of co-maintainers.

Notable projects already under the JazzBand include django-pipeline and django-configurations. The group is currently focused on Python projects and is still figuring out things like how to secure releases on PyPI.

django-configurations 1.0

Speaking of the JazzBand, members of the collective pushed out the 1.0 release of django-configurations, which is an opinionated library for writing class-based settings files for Django. The new release adds Django 1.8+ support as well as several new features.

Roundtable

The Roundtable is the home for discussions that don’t fit anywhere else.

Travis CI Sudo for Specific Environments

Next was ErikRose with an undocumented tip for Travis CI builds. As seen on the LetsEncrypt travis.yml, you can specify sudo: required for a specific entry in the build matrix to run only that build on Travis’ container-based infrastructure.

Docker on OS X via xhyve

Erik also shared xhyve, which is a lightweight OS X hypervisor. It’s a port of bhyve, and can be used as the backend for running Docker containers on OS X instead of VirtualBox. Recent changes that have made this more feasible include the removal of a 3 gigabyte RAM limit and experimental NFS support that, according to Erik, is faster than VirtualBox’s shared folder functionality. Check it out!


If you’re interested in web development at Mozilla, or want to attend next month’s Extravaganza, subscribe to the dev-webdev@lists.mozilla.org mailing list to be notified of the next meeting, and maybe send a message introducing yourself. We’d love to meet you!

See you next month!

Beer and Tell – January 2016

Once a month, web developers from across the Mozilla Project get together to talk about our side projects and drink, an occurrence we like to call “Beer and Tell”.

There’s a wiki page available with a list of the presenters, as well as links to their presentation materials. There’s also a recording available courtesy of Air Mozilla.

shobson: CSS-Only Disco Ball

First up was shobson with a cool demo of an animated disco ball made entirely with CSS. The demo uses a repeated radial gradient for the background, and linear gradients plus a border radius for the disco ball itself. The demo was made for use in shobson’s WordCamp talk about debugging CSS. A blog post with notes from the talk is available as well.

craigcook: Proton – A CSS Framework for Prototyping

Next was craigcook, who presented Proton. It’s a CSS framework that is intentionally ugly to encourage use for prototypes only. Unlike other CSS frameworks, the temptation to reuse the classes from the framework in your final page doesn’t occur, which helps avoid the presentational classes that plague sites built using a framework normally.

Proton’s website includes an overview of the layout and components provided, as well as examples of prototypes made using the framework.


If you’re interested in attending the next Beer and Tell, sign up for the dev-webdev@lists.mozilla.org mailing list. An email is sent out a week beforehand with connection details. You could even add yourself to the wiki and show off your side-project!

See you next month!

Extravaganza – January 2016

Once a month, web developers from across Mozilla get together to talk about the work that we’ve shipped, share the libraries we’re working on, meet new folks, and talk about whatever else is on our minds. It’s the Webdev Extravaganza! The meeting is open to the public; you should stop by!

You can check out the wiki page that we use to organize the meeting, or view a recording of the meeting in Air Mozilla. Or just read on for a summary!

Shipping Celebration

The shipping celebration is for anything we finished and deployed in the past month, whether it be a brand new site, an upgrade to an existing one, or even a release of a library.

Verbatim is Dead! Long Live Pontoon!

First up was Osmose (that’s me!), sharing the exciting news that Verbatim, otherwise known as localize.mozilla.org, has been decommissioned and replaced by Pontoon! Verbatim was an extremely out-of-date instance of the Pootle translation software. Mozilla websites that wish to translate their content are now encouraged to contact the Pontoon team when they want to enable the L10n community to translate their site.

Mozilla.org Dual SHA1/SHA256 Certificate Negotiation

Next was jgmize with info about www.mozilla.org‘s recent work around enabling both SHA1 and SHA256 certificates to be used on the site. Firefox is phasing out support for SHA1 certificates along with the other major browsers, but users on older browsers need to be able to download new versions of Firefox from www.mozilla.org. Some of these older versions are from before browsers supported SHA256 certificates. In order to avoid leaving these users stuck without a way to get a modern browser, www.mozilla.org needs to be able to fall back to SHA1 certificates when necessary.

Happily, www.mozilla.org is now correctly using a SHA256 certificate and falling back to a SHA1 certificate for users whose browser does not support SHA256 certificates, thanks to our CloudFlare CDN.

Peep Now Compatible with Pip 7

ErikRose shared the welcome news that Peep, the wrapper for Pip that supports hash verification of downloaded dependencies, now supports Pip 7 correctly! He also reminded us that pip 8 will obsolete Peep as it will have hash verification built-in.

Web App Developer Initiative Sites

Next was bwalker with a list of websites and tools that the Web App Developer Initiative team shipped in the last quarter:

DXR Updates

ErikRose shared a slew of DXR updates:

  • Recognizing overrides of virtual methods
  • Recognizing multiple directly overridden virtual methods
  • The ability to index Cargo packages
  • Not offering sub/superclass search when none exist

Thanks to Tom Klein and Nick Cameron for these updates!

Open-source Citizenship

Here we talk about libraries we’re maintaining and what, if anything, we need help with for them.

Product-Details Supports Python 2 and 3 and Django 1.9

Pmac shared the news that django-mozilla-product-details now supports Python 2 and 3 simultaneously, as well as supporting Django 1.9. He also shared the slightly-older news that the library supports optionally storing the data in the database instead of the filesystem.

New Hires / Interns / Volunteers / Contributors

Here we introduce any newcomers to the Webdev group, including new employees, interns, volunteers, or any other form of contributor.

  • jpetto is switching from a contractor to a full-time web developer on the Engagement Web Development team!
  • davidwalsh is moving to the Web App Developer Initiative team!
  • Osmose is moving to the Support Engineering team working on Input, SUMO, and other projects!

Roundtable

The Roundtable is the home for discussions that don’t fit anywhere else.

Docker Practices for Development

ErikRose closed us out with a few tips for using Docker for developing a website:


If you’re interested in web development at Mozilla, or want to attend next month’s Extravaganza, subscribe to the dev-webdev@lists.mozilla.org mailing list to be notified of the next meeting, and maybe send a message introducing yourself. We’d love to meet you!

See you next month!