Over the course of the last few months that has changed to being a smaller number of services that provide APIs to each other. We’ve got separate services for payments (and this), statistics (and this) and a new front end client and plans for more. The main communication mechanism between them is going to be REST APIs.

For REST APIs in Django we are currently using Tastypie, which does a pretty good job of doing a lot of things you’d need. There are a few frustrations with Tastypie and going forward I’m extremely tempted by Cornice, which we currently use for statistics.

When you ask people about consuming REST APIs in Python, lots of people tell me “we just use requests“. Requests is a great library for making HTTP calls, but when you are developing against a REST API having all the overhead of coping with HTTP is a bit much. Coping with client errors versus HTTP errors, understanding the error responses, scaling and failover and generally coping with an imperfect world.

So we took slumber and wrapped that in our our library called curling. Curling is a wrapper that makes some assumptions about the server and the client. It provides us one entry point for all our REST APIs and a place to have consistency on the client side. Below I get a timestamp on a transaction using requests.

However in the curling example it will check that one and only one result is returned (and raise meaningful errors if it didn’t) and correctly raise a meaningful error based on the HTTP response.

The result is code that is easier to write and read, but it is still relatively close to the metal of just being a REST API without being as simple as just HTTP requests or as complicated as a Web Services and SOAP stack.

As an extra bonus curling has a command line API that fills some HTTP headers out for you, syntax highlights output and the like. As a bonus, if you are calling a Django server in debug mode, rather than spew lines of HTML to the screen – it will write the HTML to a file and open a browser to the file.

These APIs are open and growing rapidly. You can find documentation on them on each project, but the main one to follow is in zamboni.

This created a few issues. The recursive pull from github was quite slow as we pulled down an awful lot of code. To update something in vendor was quite a tortuous path of updates that generated quite a few expletives from most developers the first few times they tried it. Scripts were written to make that made it easier, but that was just addressing the symptom. Multiple commits appeared in zamboni that had accidental vendor changes in the submodule and more expletives were uttered.

Because everything ended up underneath the main project anything that recursively searched directories took longer. Some git commands, greps, test runs etc took longer and longer as the vendor library grew.

Finally, building packages and using pypi and the existing community infrastructure is a good thing. It means that our code is as close to as normal as possible and our libraries are more likely to be reused if we package them properly from the start.

How we build

We don’t want any surprises about what pip is installing on our server. Often the setup.py script installs dependencies. To prevent that we invoke pip with the --no-deps flag. All dependencies have to be manually specified in the requirements files.

For each package we pin to the specific version, for example Django==1.4.3. This is a little faster since pip doesn’t have to find the relevant version. It means we have to manually update, but it ensures we don’t get a version we didn’t expect. Updates are so much simpler and easier to read in source control (for example).

Unfortunately not everything is a package. We have a few packages that pull directly from github as eggs (as of writing we have 80 python packages and 20 eggs from github). For those we set --exists-action=w to ensure that if those change the deployment continues smoothly.

Security

There are more security issues with taking a package from pypi and installing it on the production server than there is with pulling from github. We wanted to be sure that we were being as safe as possible.

So we set up a server to store local copies of our packages. It is a simple HTTP server that serves from the file system and developers have to scp packages over to it. The deployment pulls from that instead of from pypi, ensuring that we’ve got accountability and traceability about what packages are going to our production servers.

Before developers can get access to upload packages, they have to read and sign off a policy for uploading packages.

Results

The deployments are now slower because we delete the existing virtualenv and reinstall the packages from scratch each deployment. This was a temporary measure to cope with the large number of package changes over first few deployments. This part of the deployment takes about 2 minutes, which is the slowest part of the deployment. Once we stop deleting the virtualenv that should improve.

The vendor submodule still exists to look after legacy code that hasn’t been put into packages or is unlikely ever to do so. It also contains one JavaScript library.

The number of expletives issued by developers during library bumps has dropped and the number of commits with random vendor changes has vanished. Although there have been a few expletives uttered at packages. Updates are faster, git and other command line tools are faster. Overall, life is a whole lot better.

We’ve now got at least four projects using and deploying with pip and we won’t be going back. Hopefully more Mozilla web sites will now also be able to deploy with pip.

Thanks to help from Jason, Jeremy and Guillaume for making this happen.

A while back we were switching between different clusters for add-on version check as the hardware behind it changed. As we switched clusters, we also switched from a hot cache to a cold cache. This resulted in a spike in traffic:

That’s almost 45,000 requests per second coming into version check and being processed by Python and MySQL. With hardly a blip on the hardware loads, we processed that quickly. We can see the affect as the cache picked up the slack, it falls down to the usual 8,000 requests per second. Crazy traffic.

We get a large number of requests to version check for a small number of add-ons. Like Adblock Plus which, as of writing this post, has over 15 million users. That’s pretty awesome.

Looking at the traffic to Add-ons site (excluding version checks), traffic is very consistent depending upon the time of day.

So when are those times? Here’s a graph of today’s traffic that I’ve annotated.

Our low point is as India wakes up. It goes up as Europe starts to wake up and then the east coast of the US. It hits its peak at about 11am on the east coast and then as Europe and the east coast goes of line traffic starts to fall.

There’s no real surprise there and data is backed up by looking at our internal metrics that show after the United States, the biggest traffic for Add-ons comes from Germany, Russia, France in that order. But it’s always nice to be reminded of the global reach of Mozilla.

What’s common to both these cases is the weak security they employed to “safekeep” their users’ login credentials. In the case of LinkedIn, it is alleged that an unsalted SHA-1 hash was used, in the case of last.fm, the technology used is, allegedly, an even worse, unsalted MD5 hash.

Neither of the two technologies is following any sort of modern industry standard and, if they were in fact used by these companies in this fashion, exhibit a gross disregard for the protection of user data. Let’s take a look at the most obvious mistakes our protagonists made here, and then we’ll discuss the password hashing standards that Mozilla web projects routinely apply in order to mitigate these risks.

A trivial no-no: Plain-text passwords

This one’s easy: Nobody should store plain-text passwords in a database. If you do, and someone steals the data through any sort of security hole, they’ve got all your user’s plain text passwords. (That a bunch of companies still do that should make you scream and run the other way whenever you encounter it.) Our two protagonists above know that too, so they remembered that they read something about hashing somewhere at some point. “Hey, this makes our passwords look different! I am sure it’s secure! Let’s do it!”

Poor: Straight hashing

Smart mathematicians came up with something called a hashing function or “one-way function” H: password -> H(password). MD5 and SHA-1 mentioned above are examples of those. The idea is that you give this function an input (the password), and it gives you back a “hash value”. It is easy to calculate this hash value when you have the original input, but prohibitively hard to do the opposite. So we create the hash value of all passwords, and only store that. If someone steals the database, they will only have the hashes, not the passwords. And because those are hard or impossible to calculate from the hashes, the stolen data is useless.

“Great!” But wait, there’s a catch. For starters, people pick poor passwords. Write this one in stone, as it’ll be true as long as passwords exist. So a smart attacker can start with a copy of Merriam-Webster, throw in a few numbers here and there, calculate the hashes for all those words (remember, it’s easy and fast) and start comparing those hashes against the database they just stole. Because your password was “cheesecake1″, they just guessed it. Whoops! To add insult to injury, they just guessed everyone’s password who also used the same phrase, because the hashes for the same password are the same for every user.

Worse yet, you can actually buy(!) precomputed lists of straight hashes (called Rainbow Tables) for alphanumeric passwords up to about 10 characters in length. Thought “FhTsfdl31a” was a safe password? Think again.

This attack is called an offline dictionary attack and is well-known to the security community.

Even passwords taste better with salt

The standard way to deal with this is by adding a per-user salt. That’s a long, random string added to the password at hashing time: H: password -> H(password + salt). You then store salt and hash in the database, making the hash different for every user, even if they happen to use the same password. In addition, the smart attacker cannot pre-compute the hashes anymore, because they don’t know your salt. So after stealing the data, they’ll have to try every possible password for every possible user, using each user’s personal salt value.

Great! I mean it, if you use this method, you’re already scores better than our protagonists.

The 21st century: Slow hashes

But alas, there’s another catch: Generic hash functions like MD5 and SHA-1 are built to be fast. And because computers keep getting faster, millions of hashes can be calculated very very quickly, making a brute-force attack even of salted passwords more and more feasible.

So here’s what we do at Mozilla: Our WebApp Security team performed some research and set forth a set of secure coding guidelines (they are public, go check them out, I’ll wait). These guidelines suggest the use of HMAC + bcrypt as a reasonably secure password storage method.

The hashing function has two steps. First, the password is hashed with an algorithm called HMAC, together with a local salt: H: password -> HMAC(local_salt + password). The local salt is a random value that is stored only on the server, never in the database. Why is this good? If an attacker steals one of our password databases, they would need to also separately attack one of our web servers to get file access in order to discover this local salt value. If they don’t manage to pull off two successful attacks, their stolen data is largely useless.

As a second step, this hashed value (or strengthened password, as some call it) is then hashed again with a slow hashing function called bcrypt. The key point here is slow. Unlike general-purpose hash functions, bcrypt intentionally takes a relatively long time to be calculated. Unless an attacker has millions of years to spend, they won’t be able to try out a whole lot of passwords after they steal a password database. Plus, bcrypt hashes are also salted, so no two bcrypt hashes of the same password look the same.

So the whole function looks like: H: password -> bcrypt(HMAC(password, local_salt), bcrypt_salt).

We wrote a reference implementation for this for Django: django-sha2. Like all Mozilla projects, it is open source, and you are more than welcome to study, use, and contribute to it!

What about Mozilla Persona?

Funny you should mention it. Mozilla Persona (née BrowserID) is a new way for people to log in. Persona is the password specialist, and takes the burden/risk away from sites for having to worry about passwords altogether. Read more about Mozilla Persona.

So you think you’re cool and can’t be cracked? Challenge accepted!

Make no mistake: just like everybody else, we’re not invincible at Mozilla. But because we actually take our users’ data seriously, we take precautions like this to mitigate the effects of an attack, even in the unfortunate event of a successful security breach in one of our systems.

If you’re responsible for user data, so should you.

We didn’t record this Beer and Tell (my bad!), but I thought I’d share what was presented. It was pretty gaming-focused and quite cool.

Django Stats Admin

Created by Michael Kelly, this nifty little app allows you to visualize any of your Django models with datetime fields in the Django Admin. It uses the flot JS library to create pretty charts and lets you specify date ranges.

Michael’s currently working on abstracting the app out into a library that can be used with any Django app.

Lisp -> DCPU-16 Compiler

James Long, our resident Lisp champion, started off by telling us about 0x10c and the DCPU-16 assembly language in the game. It’s pretty cool stuff for gamers who are language geeks too. Of course, who wants to write machine code of any kind when there’s Lisp?! No one, that’s who. Or, well, at least not James.

James wrote a Lisp to DCPU-16 compiler that translates your Lisp to heavily optimized DCPU-16 assembly instructions. His Lisp allows inline assembly and is pretty damn slick.

James walked us threw some Lisp code, showed us the converted assembly, then ran the instructions for us! He also showed us some of the crazier stuff folks have done with the DCPU-16 language, even though it’s in its early stages. Very cool stuff.

Check out his project’s README for more info and examples. It’s actually written in James’ own Lisp variant, Outlet, that compiles to JavaScript and lua. Lisp-ception!

Even I, a steadfast vim user, was impressed.

Gladius

Dan Mosedale showed us some damn fine graphics engine fun in JavaScript, running quite smashingly (with a few browser garbage collection missteps, soon-to-be-resolved) in a browser. It was powered by a new library, Gladius.

He started with some basic objects and physics, then progressed onto the beginnings of a 3rd platformer/fighter! We discussed the current state of graphics libraries, 2D vs. 3D libraries (there are a lot of 2D libraries out there — Gladius is focusing on 3D), and gaming in browsers in general.

These folks are up to some interesting stuff and it’s a great time to get started writing games in the browser. Read more about Gladius and be sure to look for some full-fledged games using it in no time.

Apptastic, an Experiment in Cursing Creating Mobile Platforms

The developers on the Developer Ecosystem team wanted to see what it was like to develop on various mobile platforms so we’d know what to do (and what not to do) when creating a sane work flow for developing Open Web Apps. We decided to make simple Bugzilla search apps so the each experiment had a common goal.

Turns out: most mobile platforms are a pretty big pain in the ass to develop for! It’s important to remember that we’ve come a long way (before the iPhone, it was sell your “app” through each carrier to a specific device for a crazy price), but we still have a ways to go.

Arron wrote a pretty sweet write-up of his Windows Phone experience, and we briefly discussed developing on other platforms, including the bureaucracy involved in iOS development and the general pain of Android… every step of the way.

Look for more blog posts about our experiences in the future.

That’s All Folks!

Hope you enjoyed my recap of the Beer and Tell. We’ll be sure to record it for next time!

Until then: keep hacking and having fun!

One thing that I’ve wanted to add is information on when deployments occur as these are often critical periods that can trigger changes in the graphs. Sure enough, this has all been done before and covered in a blog post. Following on from that blog post, here’s how I added this for the Marketplace.

First, I added a management command to django-statsd that allows you to easily send a ping to Graphite for a specific key. The value of the key is set to be a UNIX time stamp. Once that was in place I altered our deployment scripts so that on each update they send a ping with the key “update”:

python2.6 manage.py statsd_ping --key=update

Then each time a server is deployed that key gets added to Graphite.

Next, I altered our graphs so that they show this ping as a vertical line. That’s using the command drawAsInfinite, for example:

drawAsInfinite(stats.timers.addons-dev.update.count)

Our development server updates on every commit. In other words, it continually deploys all the time. This is the graph for the development server, and you can see the large number of small commits we had this afternoon (each commit is indicated by a pink line):

A more realistic view is our production server, where you can see our Thursday afternoon push in context (each push is a pink line):

Adding this data is crucial to helping the Marketplace get closer to continuous deployment in the coming months.

Enter Scrum

Being software developers, many of us know and like Scrum. I’ve personally found it to be a great method of answering the old questions of “How long will this take?”, and “How much can we get done?”. I won’t go into great detail on how Scrum works here, but the main points are as follows:

• Work units are scored by the team based on difficulty or expected time consumption.
• Those work units are included in a “Sprint” usually lasting 2 weeks.
• Lots of interesting data is generated around team performance.

That last point is key. In order to be able to use and appreciate these data, we need to be able to collect and visualize them in a useful way. This normally means awesome charts and graphs. As nice as Bugzilla is for coordination and issue tracking, it has an unfortunate lack of awesome data visualization tools. It does however, have an API.

The Birth of Scrumbu.gs

There have been some valiant efforts at improving the interface of Bugzilla. One of these, BugzillaJS, even adds some pie charts for Scrum data in a bug-list. But what I really wanted was a burndown.

The data you need to associate with a bug (or story in Scrum parlance) are the user who will benefit from the work, the component of the product to be affected, and a score (points) representing the perceived difficulty of the work. Bugzilla has no fields for these out of the box, but a catchall field called the “whiteboard” can be formatted in such a way as to allow tools like BugzillaJS and Scrumbu.gs to easily find it (see below). This is obviously not optimal, but it’s what we have for the moment. And when you combine that with the bug history data from the Bugzilla API, you have all you need to calculate burndowns, team and developer velocities, and all sorts of even awesomer things.

The result of this need, my excellent bosses allowing me the time to experiment with cool stuff, and the existence of Python, Django, Slumber (REST client), Twitter Bootstrap, jQuery, Flot, and Epio (python/django hosting), is Scrumbu.gs. It’s still a work in progress, but you can click through some existing projects and sprints to see how it’s working so far. The code and issue tracker are on github, so feel free to poke around and see how it works. If you see a problem or have an idea for a feature, please file an issue, and if you have the ability and/or desire, send a pull-request .

I recently gave a lightning talk about the project for my team at Mozilla and I’ve included it below if you’re interested. My hope with this project was to spread the use of Scrum around Mozilla and other teams who use Bugzilla, and to improve the tools for those who already do. If your team is interested in tracking your sprints using this tool, participating in its development, or just chatting about Scrum, please join us in the #scrum channel on IRC.

Download video as MP4, WebM, or Ogg.

Edit 3/28: Added links to Slumber, Epio, and jQuery.

Along with a few other Mozilla sites, we’ve been using pystatsd and Graphite for a while. This produces useful graphs of the site health and performance. For example, this graph plots HTTP responses we serve out from Python (this is not the amount of traffic to our site which is much larger).

To facilitate using the timing API we wrote a module between statsd and Django called django-statsd that, amongst other things, provides a way to interface between the browser and Python backend. The library has support for boomerang, a great front end timing library. I quickly wrote a much simpler service called stick, which sends a small subset of timings that we want.

Using stick is straightforward – once you’ve setup your Python backed and included your JavaScript, you call it in your page. For example:

stick.send('https://to.your.site/record')

The API returns with timestamps for each of the events. To give us meaningful numbers, we then subtract the timestamp for each of the events from navigationStart so we can tell the timings relative to that start point. And here’s the end result in our Graphite server:

For internal sites we collect 100% of the data. However, on our live site we get quite a lot of traffic, so we’ve rolled this out by collecting only 1% of traffic. We’ll start increasing this percentage once we are confident this won’t be causing problems. We’ll also be adding it to the about:addons page which gets quite a bit of traffic.

We’ll be tweaking things to ensure we pass through the right numbers and interpret them correctly. The amount of data is probably too low to be useful, right now, but I can’t help but wonder what the peak around 12.30pm today was…

You’ll end up with production data in your Django database and that will likely contain different kinds of data such as: configuration data, required basic data (categories for example), collected data and personal user data. There’s a couple of reasons for taking that production data and copying it off your production servers:

• for developers and contributors you want a sample copy of the app with some key data in.
• for testing or staging servers, you might want to copy down the database from the production server so you can test certain scenarios or load.

Extracting parts of your database

For the first case, it’s nice to prepare a minimal copy of the database that contains key data. For example, for those wanting to develop or contribute to addons.mozilla.org we have Landfill by Wil Clouser.

Django comes with a nice facility for loading data, fixture dumping and loading. This can be used to pull data out of your database and then reload it. However the built in Django dumpdata dumps all the records for your model (depending upon your default object manager). That might not be what you want for this scenario. So a useful utility for dumping just the records you want is provided Django fixture magic written by Dave Dash.

A standard dumpdata looks like this:

manage.py dumpdata users.UserProfile

And will dump every UserProfile. By contrast:

manage.py dump_object users.UserProfile 1

Will just dump the UserProfile with primary key of 1. Django fixture magic also has a few other useful things such as merging and reordering fixtures.

This allows you to trim a set of fixtures from your live database down quickly. Then developers or contributors can load the key parts of the database that they need from those fixtures.

Anonymising the database

Sending the production database downstream to internal developers or internal test sites is a pretty common use case. This process does not require a complete clean of the database, but it does require some cleaning of database. If you stored credit card data, for example, you’d never want to copy that off your production database.

At Mozilla we use an anonymising script, written by Dave Dash again. There are few options: to truncate, nullify, randomize or selectively delete. The format is a simple YAML file, for example:

   tables:
        users:
            random_email: email
            nullify:
                - firstname

This is a snippet from the config script for addons.mozilla.org.

When the IT copies the databases down from production, this script is run against the database. Ensuring that when us developers access the backups to investigate certain issues, we’ll be getting the bits we want and not the bits that might expose user data.

In the next blog post we’ll look at logs and tracebacks.

A lot of people focus on open sourcing their Django libraries, but at Mozilla we open source the entire site. Releasing your entire source code can lead to a few problems, firstly let’s look at your Django settings file.

Separating your settings

All Django sites come with a settings.py file. This file contains some key settings that you should not be releasing to the general public, such as database configuration and the secret key. The simple way to do this is have another file that contains the secret settings and then import it from settings.py. For example include this at the bottom of your settings.py:

All your sensitive settings can now kept in that local file on your server and should not be published as part of your site code. This file will override the base settings file.

There are plenty of other examples on different ways to do this. You can do it in manage.py, or turn your settings.py into a folder that Python can import. Make sure that you ignore your settings_local.py file in your source control.

If you add in new settings, make sure you add them into the main settings.py file. Even if they are just empty strings, lists or whatever, it will mean that when you call settings.SOME_KEY in your code, you won’t have to cope with the setting not being present. There’s nothing more tedious than writing lots of getattr code to cope with that.

Viewing settings on the server

One downside of doing this is that you might not be sure what your settings are on the server. At Mozilla only the system administrators who manage and deploy our servers can see the contents of that file. But it’s still helpful to check the settings on the server. For that we wrote a settings page that lists them out.

Django helps by providing a method that lists the settings, but obscures those really sensitive values:

On addons.mozilla.org we require an account to have certain privileges before showing the page. But even if that did get broken into, you wouldn’t know our SECRET_KEY or anything very useful. Here’s how that page looks:

Now that you’ve got your settings files ready, you can confidently open source your Django project safe that you won’t be leaking any key data.

In the next blog post we’ll look at scrubbing personal data from your database.