Extravaganza – August 2016

Once a month, web developers from across Mozilla get together to talk about the work that we’ve shipped, share the libraries we’re working on, meet new folks, and talk about whatever else is on our minds. It’s the Webdev Extravaganza! The meeting is open to the public; you should stop by!

You can check out the wiki page that we use to organize the meeting, or view a recording of the meeting in Air Mozilla. Or just read on for a summary!

Shipping Celebration

The shipping celebration is for anything we finished and deployed in the past month, whether it be a brand new site, an upgrade to an existing one, or even a release of a library.

Normandy Control Interface Release

First up was mythmon, who mentioned the new release of Normandy, the backend server for the SHIELD system that powers surveys and feature studies in Firefox. This release includes a new admin interface built using React and Redux, as well as a switch to client-side targeting that is powered by JEXL expressions.

Google Sign-In on Socorro

Next was peterbe, who mentioned that Socorro, the crash-report service for Firefox, has added Google-based sign-in ahead of the planned shut-down of Persona. It’s planned to land in production sometime within the next week, and involves some extra work around triggering automatic sign-out of users who have been signed in for a certain amount of time.

DXR: The Ballad of Peter Elmers

ErikRose was next, and shared yet another list of new features developed by DXR intern new_one:

  • Description column in file listings
  • Better handling of whitespace in paths
  • Modification dates are pulled from the VCS instead of the filesystem
  • Per-line blame links
  • Badges in the filter dropdown showing what languages support each filter

Open-source Citizenship

Here we talk about libraries we’re maintaining and what, if anything, we need help with for them.

django-jinja-markdown

Next was pmac in absentia, who wanted to share django-jinja-markdown, a fork of jingo-markdown. It adds support for rendering Markdown strings to HTML in templates rendered with django-jinja via a markdown filter, as well as a similarly-named template function. It also includes a block-level template tag that can be enabled by adding the library as a Jinja extension.

json-schema-reducer

Back to peterbe, who shared json-schema-reducer. The Python-based library takes in a JSON Schema and a JSON object or dict, and returns the JSON object with only fields that are specified in the schema. The main use case for the library is taking Socorro crash reports, and whitelisting data that is appropriate to be sent to Mozilla’s Telemetry platform for analysis, removing sensitive data that isn’t meant to leave the crash report system.

Roundtable

The Roundtable is the home for discussions that don’t fit anywhere else.

GTD

Last up was ErikRose, who brought up the Getting Things Done methodology and how he recently has adopted it to help deal with his personal and professional time management. The video recording contains an extended discussion of time management strategies, but useful tools highlighted during the discussion include Things (OSX only), Org-Mode, and good old-fashioned sticky notes.


If you’re interested in web development at Mozilla, or want to attend next month’s Extravaganza, subscribe to the dev-webdev@lists.mozilla.org mailing list to be notified of the next meeting, and maybe send a message introducing yourself. We’d love to meet you!

See you next month!

Beer and Tell – July 2016

Once a month, web developers from across the Mozilla Project get together to talk about our side projects and drink, an occurrence we like to call “Beer and Tell”.

There’s a wiki page available with a list of the presenters, as well as links to their presentation materials. There’s also a recording available courtesy of Air Mozilla.

Moby von Briesen: Jam Circle

This week’s only presenter was mobyvb, who shared Jam Circle, a webapp that lets users play music together. Users who connect join a shared room and see each other as circles connected to a central node. Using the keyboard (or, in browsers that support it, any MIDI-capable device), users can play notes that all other users in the channel hear and see as colored lines on each circle’s connection to the center.

The webapp also includes the beginnings of an editor that will allow users to write chord progressions and play them alongside live playback.

A instance of the site is up and running at jam-circle.herokuapp.com. Check it out!


If you’re interested in attending the next Beer and Tell, sign up for the dev-webdev@lists.mozilla.org mailing list. An email is sent out a week beforehand with connection details. You could even add yourself to the wiki and show off your side-project!

See you next month!

Extravaganza – July 2016

Once a month, web developers from across Mozilla get together to talk about the work that we’ve shipped, share the libraries we’re working on, meet new folks, and talk about whatever else is on our minds. It’s the Webdev Extravaganza! The meeting is open to the public; you should stop by!

You can check out the wiki page that we use to organize the meeting, or view a recording of the meeting in Air Mozilla. Or just read on for a summary!

Shipping Celebration

The shipping celebration is for anything we finished and deployed in the past month, whether it be a brand new site, an upgrade to an existing one, or even a release of a library.

Basket switch to Salesforce

First up was pmac, who shared the news that Basket, email newsletter subscription service, has switched to using Salesforce as the backend for storing newsletter subscriptions. In addition, the service now has a nifty public DataDog metrics dashboard showing off statistics about how the service is performing.

Engagement Engineering Status Board

Next was giorgos, who shared status.mozmar.org, a status page listing the current status of all the services that Engagement Engineering maintains. The status board pulls monitoring information from Dead Man’s Snitch as well as New Relic‘s application and Synthetics monitoring. The app runs a worker using AWS Lambda that pulls the information and writes it to a YAML file in the repo‘s gh-pages branch, and the status page itself reads the YAML file via JavaScript to build the display.

DXR

ErikRose stopped by to share more cool things that shipped in DXR this month:

  • Indexing for XBL and JavaScript.
  • Indexing 32+ new projects
  • Added a 3rd build server
  • Several performance optimizations that cut down build times by roughly 25%.
  • C++ macro definitions, method overrides, pure virtuals, substructs, and more are all now indexed. In addition, you can now easily jump between header files and their implementations.
  • UI improvements, including contrast improvements, a new filename filter, and jumping directly to files that are the only result of a query.

Special thanks to intern new_one and contributors twointofive and abbeyj. Also special thanks to MXR for being shut down due to security bugs and allowing DXR to flourish in its wake.

Fathom 1.0 and 1.1

Erik also brought up Fathom, an experimental framework for extracting meaning from webpages. Fathom allows you to write declarative rules that score and classify DOM nodes, and then extract those nodes from a DOM that it analyzes.

This month we shipped the 1.0 version of Fathom, as well as a 1.1 release with a bug fix for Firefox support as well as an optimization fix. It’s available as an NPM module for use as a library.

Roundtable

The Roundtable is the home for discussions that don’t fit anywhere else.

Engagement Engineering Hiring – Senior Webdev and Site Reliability Engineer

Last up was pmac again, who wanted to mention that the Mozilla Engagement Engineering team is hiring a Senior Web Developer and a Site Reliability Engineer. If you’re interested in working at Mozilla, click those links to apply on our careers site!


If you’re interested in web development at Mozilla, or want to attend next month’s Extravaganza, subscribe to the dev-webdev@lists.mozilla.org mailing list to be notified of the next meeting, and maybe send a message introducing yourself. We’d love to meet you!

See you next month!

Auto-generating a changelog from git history

At Mozilla we share a lot of open source libraries. When you’re using someone else’s library, you might find that an upgrade breaks something in your application — but why? Glancing at the library’s changelog can help. However, manually maintaining a changelog when building features for a library can be a challenge.

We’ve been experimenting with auto-generating a changelog from commit history itself and so far it makes changelogs easy and painless. Here’s how to set it up. These tools require NodeJS so it’s best suited for JavaScript libraries.

First, you need to write commit messages in a way that allows you to extract metadata for a changelog. We use the Angular conventions which specify simple prefixes like feat: for new features and fix: for bug fixes. Here’s an example of a commit message that adds a new feature:

feat: Added a `--timeout` option to the `run` command

Here’s an example of a bug fix:

fix: Fixed `TypeError: runner is undefined` in the `run` command

The nice thing about this convention is that tools such as Greenkeeper, which sends pull requests for dependency updates, already support it.

The first problem with this is a social one; all your contributors need to follow the convention. We chose to solve this with automation by making the tests fail if they don’t follow the conventions 🙂 It’s also documented in our CONTRIBUTING.md file. We use the conventional-changelog-lint command as part of our continuous integration to trigger a test failure:

conventional-changelog-lint --from master

There was one gotcha in that TravisCI only does a shallow clone which doesn’t create a master branch. This will probably be fixed in the linter soon but until then we had to add this to our .travis.yml:

Alright, everybody is writing semantic commits now! We can generate a changelog before each release using the conventional-changelog tool. Since we adopted the Angular conventions, we run it like this before tagging to get the unreleased changes:

conventional-changelog -p angular -u

This scrapes our commit log, ignores merges, ignores chores (such as dependency updates), ignores documentation updates, and makes a Markdown list of features and fixes linked to their git commit. Example:

### Bug fixes
* Fixed `TypeError: runner is undefined` in the `run` command ([abc1abcd](https://github.com/.../))

### Features
* Added a `--timeout` option to the `run` command ([abc1abcd](https://github.com/.../))

As you can see, we also make sure to write commit messages in past tense so that it reads more naturally as a historic changelog. You can always edit the auto-generated changelog to make it more readable though.

The conventional-changelog tool can update a README.md file but, for us, we just paste the Markdown into our github releases so that it shows up next to each release tag.

That’s it! There are a lot of options in the tools to customize linting commits or changelog generation.

Extravaganza – June 2016

Once a month, web developers from across Mozilla get together to talk about the work that we’ve shipped, share the libraries we’re working on, meet new folks, and talk about whatever else is on our minds. It’s the Webdev Extravaganza! The meeting is open to the public; you should stop by!

You can check out the wiki page that we use to organize the meeting, or view a recording of the meeting in Air Mozilla. Or just read on for a summary!

Shipping Celebration

The shipping celebration is for anything we finished and deployed in the past month, whether it be a brand new site, an upgrade to an existing one, or even a release of a library.

Okta SSO Fix Add-on

First up was shobson, who shared MoCo SSO Tweaks, a Firefox add-on that makes the Okta SSO flow better. Among other things, it auto-focuses the verification code field, makes Enter submit the verification code form, and optionally automatically checks the “Remember Device” checkbox.

ViewSourceConf.org

Next was hoosteeno, who shared viewsourceconf.org, the website for the 2016 edition of View Source, a conference for the web that Mozilla runs. The site is statically generated by Metalsmith and takes advantage of several plugins include the model plugin and the template plugin for Swig templates.

shobson worked on the frontend for the site. It intentionally avoids using jQuery and uses SVGs for small image filesizes and easy Retina support. In the future the site will be using Service Workers to enable offline access for the site, and particularly the schedule.

Air Mozilla Timenails

peterbe stopped by to talk about the timenail support he added to Air Mozilla. A timenail is a single screenshot from a video at a specific timestamp; a series of timenails is generated by the transcoding process and made available in the chapter editing interface for each video to help aid in finding good transition points for marking chapters in the video.

The site also calculates the difference between timenails and lets users filter the timenail list to only show thumbnails that have a difference between the previous timenail above a given threshold.

DXR

Next up was ErikRose, sharing a bunch of new changes that landed in DXR:

  • Case-sensitivity is now inferred based on whether the search text is mixed case or not.
  • Support for XPIDL
  • 20% lower memory use
  • Improved C++ analysis, including template support.

Thanks to new_one and Tom Klein for contributing these improvements!

Peep 3.1.2

Erik also mentioned that peep 3.1.2 is out, with support for pip 8.1.2.

Fathom

Erik’s last mention was Fathom, which is an experimental framework for extracting meaning from webpages. You provide it declarative rules that score and classify DOM nodes, and in return it will parse a webpage and rank DOM nodes in the page based on the given rules. You can then extract nodes from this ranking in various ways, such as finding the highest-ranked nodes for a specific attribute, or finding clusters of similarly-classified nodes that are close to each other.

Readable Bug Statuses in Bugzilla

emceeaich and dylan wanted to share (in absentia) the news that the Readable Bug Status package that emceeaich has been working on has been deployed on bugzilla.mozilla.org for bugs in the Firefox, Core, Toolkit, and BMO products. Readable Bug Status helps summarize the status of a bug using information from several fields.

Bedrock + Gulp

Next was pmac who shared the news that Bedrock has switched to using gulp to help manage their frontend build and development process. The site previously relied on django-pipeline completely to manage its static assets, but ran into issues with slow builds during development, as django-pipeline doesn’t watch for which files have changed and simply triggers a full build on each pageview. Switching to gulp allows the site to only rebuild frontend files that have changed, and it builds them as soon as they change instead of building them on-demand.

For more details, pmac wrote a blog post describing the switch.

Open-source Citizenship

Here we talk about libraries we’re maintaining and what, if anything, we need help with for them.

Google Analytics Pageviews on Non-Web Requests

peterbe shared a blog post he wrote describing how to use Raven to send pageview data to Google Analytics. This is useful for tracking usage of endpoints that don’t return webpages but are still considered part of your site’s public API.

Domainswitcher

peterbe also shared Domainswitcher, and add-on that lets you easily switch between domains while preserving the current path. The add-on is useful for web developers working on sites that want to switch between prod, staging, development, and locally-hosted instances of the site.

New Hires / Interns / Volunteers / Contributors

Here we introduce any newcomers to the Webdev group, including new employees, interns, volunteers, or any other form of contributor.

Name Role Work
Benton Case Intern – Web Developer Add-on Recommendation

If you’re interested in web development at Mozilla, or want to attend next month’s Extravaganza, subscribe to the dev-webdev@lists.mozilla.org mailing list to be notified of the next meeting, and maybe send a message introducing yourself. We’d love to meet you!

See you next month!

Django, Pipeline, and Gulp

Bedrock, the code behind www.mozilla.org, is a very large Django project. It is mostly large due to the volume and diversity of independent pages it serves. These pages come with a surprising amount of static media (css, js, images, fonts, etc.). So, any system that we use to deal with said media should be efficient in order to keep our development servers fast.

We like django-pipeline for managing our static media in production. It does a great job of bundling, minifying, and compressing our css and js. When using it in a development environment however, it does not scale well. The issue is that it does not watch for changes to your files, so all it can do is copy them all from their source to the static directory on every page load. For a reasonable number of files this is probably fine, but as I said ours is not that. This is exasperated in slow I/O environments like Docker on a non-linux system (like OSX). We’ve not been able to setup an acceptable Docker-based local dev environment yet because it can literally take several minutes to render the home page.

Due to all of the issues noted above we’ve been looking for other ways of handling static media. We’ve considered a few times to move to a completely nodejs based system that would be completely independent of the Django side, and may still do that some day, but the problem has always been scope and impact. Again because the project is so large, making sweeping changes that affect workflow and all static files can both take a lot of time and be very disruptive when they land. So for a long time we figured we were stuck. But recently a conversation started in IRC about being able to just disable django-pipeline’s copying of files. If we could do that we could use gulp-watch to much more quickly and efficiently manage these files while being edited and still get the benefits of django-pipline for production. It turned out that someone else already had this idea and mostly we just needed to upgrade django-pipeline.

After that it was a simple matter of adding a task to our Gulpfile:

gulp.task('media:watch', function () {
    return gulp.src('./media/**/*')
        .pipe(watch('./media/**/*', {
            'verbose': true
        }))
        .pipe(gulp.dest('./static'));
});

But it was still a bit odd now having to have two shells open, one for the gulp task and another for the Django dev server. So we did a little more gulp magic and now have a single command to start up both the file watching and the Django server that combines the output of both in a single terminal.

gulp.task('serve:backend', function () {
    var devServerPort = process.env.PORT || 8000;
    process.env.PYTHONUNBUFFERED = 1;
    process.env.PYTHONDONTWRITEBITECODE = 1;
    spawn('python', ['manage.py', 'runserver', '0.0.0.0:' + devServerPort], {
        stdio: 'inherit'
    });
});

gulp.task('default', function() {
    gulp.start('serve:backend');
    gulp.start('media:watch');
});

You can see the full gulpfile.js in our repo on Github if you’d like. It’s a simple but very effective change and has made a very noticeable improvement in the speed of the local development server. And now we hope that we can finally complete and recommend Docker as the default setup for local development on bedrock.

Extravaganza – May 2016

Once a month, web developers from across Mozilla get together to talk about the work that we’ve shipped, share the libraries we’re working on, meet new folks, and talk about whatever else is on our minds. It’s the Webdev Extravaganza! The meeting is open to the public; you should stop by!

You can check out the wiki page that we use to organize the meeting, or view a recording of the meeting in Air Mozilla. Or just read on for a summary!

Shipping Celebration

The shipping celebration is for anything we finished and deployed in the past month, whether it be a brand new site, an upgrade to an existing one, or even a release of a library.

Normandy, the Recipe Server

First up was Osmose (that’s me!), sharing the news that Normandy has shipped! Normandy is a service that will eventually power several Firefox features that involve interacting with users and testing changes to Firefox quickly and safely, such as recommending features that may be useful to users or offering opportunities to try out changes. Right now the service is powering Heartbeat surveys being sent to release users.

Big thanks to the User Advocacy and Web Engineering teams for working on the project!

MDN Save Draft Feature

Next was shobson who talked about MDN‘s Safe Draft feature. When editing an MDN article, the site autosaves your edits to localStorage (if it’s available). Then, when you revisit the editing interface later, the site offers to let you restore or discard the draft, disabling autosave until a decision is made. Future improvements may include previewing drafts and notifying users when an article has changed since their draft was saved.

Air Mozilla Thumbnails

peterbe stopped by to talk about Air Mozilla‘s chapters feature, which allows users to mark and link to segments in a video. The site now auto-generates thumbnails for chapters to help preview what the chapter is about.

Roundtable

The Roundtable is the home for discussions that don’t fit anywhere else.

Docker Development Environments

Last up was jgmize, who asked about use of Docker for easy development environments. The general consensus was that most of the developers present had tried using Dockerized development environments, but tended towards using it only for deployed services or not at all.

Some of the interesting projects brought up for using Docker for development or deployment were:

Check ’em out!


If you’re interested in web development at Mozilla, or want to attend next month’s Extravaganza, subscribe to the dev-webdev@lists.mozilla.org mailing list to be notified of the next meeting, and maybe send a message introducing yourself. We’d love to meet you!

See you next month!

Beer and Tell – April 2016

Once a month, web developers from across the Mozilla Project get together to talk about our side projects and drink, an occurrence we like to call “Beer and Tell”.

There’s a wiki page available with a list of the presenters, as well as links to their presentation materials. There’s also a recording available courtesy of Air Mozilla.

emceeaich: Memory Landscapes

First up was emceeaich, who shared Memory Landscapes, a visual memoir of the life and career of artist and photographer Laurie Toby Edison. The project is presented as a non-linear collection of photographs, in contrast to the traditionally linear format of memoirs. The feature that emceeaich demoed was “Going to Brooklyn”, which gives any link a 1/5 chance of showing shadow pictures briefly before moving on to the linked photo.

lorchard: DIY Keyboard Prototype

Next was lorchard, who talked about the process of making a DIY keyboard using web-based tools. He used keyboard-layout-editor.com to generate a layout serialized in JSON, and then used Plate & Case Builder to generate a CAD file for use with a laser cutter.

A flickr album is available with photos of the process.

lorchard: Jupyter Notebooks in Space

lorchard also shared eve-market-fun, a Node.js-based service that pulls data from the EVE Online API and pre-digests useful information about it. He then uses a Jupyter notebook to pull data from the API and analyze it to guide his market activities in the game. Neat!

Pomax: React Circle-Tree Visualizer

Pomax was up next with a new React component: react-circletree! It depicts a tree structure using segmented concentric circles. The component is very configurable and can by styled with CSS as it is generated via SVG. While built as a side-project, the component can be seen in use on the Web Literacy Framework website.

Pomax: HTML5 Mahjong

Also presented by Pomax was an HTML5 multiplayer Mahjong game. It allows four players to play the classic Chinese game online by using socket.io and a Node.js server to connect the players. The frontend is built using React and Webpack.

groovecoder and John Dungan: Codesy

Last up was groovecoder and John Dungan, who shared codesy, an open-source startup addressing the problem of compensation for fixing bugs in open-source software. They provide a browser extension that allows users to bid on bugs as well as name their price for fixing a bug. Users may then provide proof that they fixed a bug, and once it is approved by the bidders, they receive a payout.


If you’re interested in attending the next Beer and Tell, sign up for the dev-webdev@lists.mozilla.org mailing list. An email is sent out a week beforehand with connection details. You could even add yourself to the wiki and show off your side-project!

See you next month!

Extravaganza – April 2016

Once a month, web developers from across Mozilla get together to talk about the work that we’ve shipped, share the libraries we’re working on, meet new folks, and talk about whatever else is on our minds. It’s the Webdev Extravaganza! The meeting is open to the public; you should stop by!

You can check out the wiki page that we use to organize the meeting, or view a recording of the meeting in Air Mozilla. Or just read on for a summary!

Shipping Celebration

The shipping celebration is for anything we finished and deployed in the past month, whether it be a brand new site, an upgrade to an existing one, or even a release of a library.

A-Frame v0.2.0

First up was ngoke, who shared the news that A-Frame shipped a new release! The v0.2.0 release focuses on extensibility by improving the Component API and adding a slew of new objects that can be registered for use in the framework. A blog post is available that explains the changes.

Let’s Encrypt Switches to Pip 8

ErikRose wasn’t able to attend, but left notes on the wiki about Let’s Encrypt switching from peep to pip 8 for installing its requirements.

DXR Indexing Most Mozilla-Central Branches

Erik also left a note mentioning that DXR, Mozilla’s code browser and search tool for the Firefox codebase, now indexes more branches from mozilla-central, including Aurora, Beta, Central, ESR45, and Release.

Pipstrap 1.1.1

The last note that Erik left mentioned that Pipstrap, a small script for bootstrapping a hash-checked version of Pip, shipped a new release that fixes an error reporting bug under Python 2.6.

Snippets in AWS + Deis

Next up was bensternthal, who shared the news that snippets.mozilla.com is now hosted on Engagement Engineering’s Deis cluster in AWS. Thanks to giorgos, jgmize, and pmac for helping to move the site over! Ben also mentioned that Basket is the next service slated to move over.

Open-source Citizenship

Here we talk about libraries we’re maintaining and what, if anything, we need help with for them.

New django-csp Maintainer

pmac stopped by to let us know that he is taking over maintenance of django-csp. Thanks to jsocol for authoring and maintaining the library until now!

Hashin Python Version Filtering

Last up was peterbe, who shared news of a new release of hashin. The main addition is a --python-version flag, which allows you to filter the hashes hashin adds by python versions that you plan to use, reducing the number hashes added. Thanks to mythmon for submitting the patch!


If you’re interested in web development at Mozilla, or want to attend next month’s Extravaganza, subscribe to the dev-webdev@lists.mozilla.org mailing list to be notified of the next meeting, and maybe send a message introducing yourself. We’d love to meet you!

See you next month!

Beer and Tell – March 2016

Once a month, web developers from across the Mozilla Project get together to talk about our side projects and drink, an occurrence we like to call “Beer and Tell”.

There’s a wiki page available with a list of the presenters, as well as links to their presentation materials. There’s also a recording available courtesy of Air Mozilla.

Osmose: Mandelbrot in Rust

First up was Osmose, who showed off a Rust program he “wrote” that generates a PNG of the Mandelbrot set. And by “wrote”, he means he copied it from an in-progress book on Rust called “Programming Rust” by fellow Mozillian jimb. The book uses the Mandelbrot example to illustrate how easy writing safe parallel code in Rust is. The book is available as an in-progress Early Release from O’Reilly, and is aimed towards people with prior programming experience, especially in another systems programming language.

Peterbe: PodcastTime

Next was peterbe, show shared PodcastTime, a directory of over 13 years of podcast metadata that shows you how much time per day, week, and month you need to spend listening to your podcasts to keep up with them. The site also shows a calendar of upcoming and past podcasts in your list, and lets you share your list of podcasts with others.


If you’re interested in attending the next Beer and Tell, sign up for the dev-webdev@lists.mozilla.org mailing list. An email is sent out a week beforehand with connection details. You could even add yourself to the wiki and show off your side-project!

See you next month!