I’m delighted to see that the new code is roughly 20x faster than the previous version, without using processor-specific non-portable assembly language. The old “ref” code, on my 2008 laptop (2.53GHz Core2Duo), makes signatures in 2ms and verifies them in 7ms. The “ref10″ code signs in 120us and verifies in 307us. That’s over 8300 signature per second! The ref10 version also includes the batch-verification function, which (thanks to some tricks in the design of Ed25519) makes it faster to verify many signatures at once. Interestingly, this requires random numbers on the *verification* side (since it’s doing statistical verification: if the attacker knew which random numbers you were going to use, they could craft a set of message that would appear valid when checked by the batch verifier, but were invalid when checked individually).

Naturally, this release came exactly one day after I finally published python-ed25519 1.0 . But 1.1 will have the speedups.

• where should it be stored, so that other code can find it. PEP 8 tells us to use __version__, and distutils tells us to call setup() with a version= argument. The embedded string is particularly useful to report or record a version in bug reports.
• what format it should take: PEP 386 describes a format (N.N[.N]+[{a|b|c|rc}N[.N]+][.postN][.devN]) that enables comparison, so packaging tools can evaluate things like “dependency > 1.2.0″. (I happen to find this format really limiting, and this tool doesn’t necessarily produce PEP386-compliant strings, but that’s not what this post is about)

What I do mean is:

• how does the right version string get into the code?
• what does a release manager need to do when it’s time to make a new release?

The traditional approach, ages old, is to have a static string embedded in the code. Each time you’re about to make a new release, you make a commit which updates this string. It’s nice and simple, but has some problems:

• Confusing intermediate versions: users/developers who work from revision control, between releases, will get a stale version, causing confusing bug reports (“it says 1.2! Oh, but I pulled from SVN last tuesday, maybe it’s closer to 1.3″). I got into the habit of commiting a “1.2+” version just after making the 1.2 release, which fixes the confusion, but doesn’t remove the ambiguity (“it says 1.2+ .. when did you last update?”)
• Post-RC code changes: projects large enough to have release candidates must accept (small) code changes between the last release-candidate and the final release, to update the embedded version string one last time. This makes QA nervous.
• Merge conflicts: embedded version strings work fine for linear histories, but modern DVCS systems make it easy to have extended release branches, so larger projects usually have parallel lines of development (dev, stabilization, release). It’s frequently a good idea to merge your release branch back into your main trunk, to make sure you don’t lose any of the fixes made during the release effort. But then you get merge conflicts between the version string on trunk (which now says something like 1.3-dev or 1.3+) and the older version on the release branch (maybe 1.2), every single release. Resolving this correctly each time (generally in favor of the trunk version) is a source of errors.
• It adds an extra step to the release process (two if you use the “1.2+” approach), discouraging developers from releasing early and often.

Over the years, I’ve built a couple of ad-hoc mechanisms for updating the embedded version string just before each commit, or to add an “update version” command/script that checks with the version-control tool for information. Tahoe uses setup.cfg to prepend an update-versions command in front of any setup.py command that might care. And I’ve had setup.py code which greps a _version.py for the current string (rather than import, to avoid committing to an unusably old dependency, also for speed). But none of these have been very satisfying.

Thinking about how I use git these days, I realized that I want my release process to have one step: “git tag” (well, and a “git push” to tell the world about it). Everything else should be automated: building tarballs, uploading them to a release server, updating a web page, sending an announcement email, pypi registration, etc. What really matters is the release manager making the decision to bless some well-tested revision id with a public name of some sort.

So I finally built a tool to accomplish this. It’s called “Versioneer”, and is available at https://github.com/warner/python-versioneer . It’s still pretty early, but seems to do the right thing. Here’s how it works:

• To install, you copy versioneer.py into your tree and follow the instructions in the docstring. These will have you edit your setup.py to “import versioneer”, set a few variables to tell it about your tag-naming convention (PROJECT-VER or just VER) and where the _version.py ought to live. It also has you add a cmdclass= to your setup() arguments, which hooks into the “build” and “sdist” commands, and adds an “update_files” command. Then you run “setup.py update_files”, which creates your _version.py and modifies your __init__.py to include it (and define the PEP 8 __version__ variable). It also creates a .gitattributes file to arrange for variable-expansion during git-archive, described below. Then it does ‘git add’ on the relevant files and asks you to commit the changes.
• When setup.py needs a version (such as for building an sdist tarball, or registering with pypi), it calls versioneer.get_versions().
• At runtime, __init__.py calls _version.get_versions() to populate __version__
• If either is called in a checked out source tree, they invoke “git describe” (with –tags –always –dirty) to come up with a fine-grained version string. If you’re sitting on a tag, you get just “1.4″. If you’re after a tag, you’ll get something like “1.4-8-gf7283c2″, which means there are 8 commits after the 1.4 tag, and the abbreviated SHA1 revision ID is f7283c2. And if your tree has uncommitted changes, you’ll get “1.4-8-gf7283c2-dirty”.
• Using “git archive” to create a tarball or zipfile will expand some magic variables in _version.py, capturing the SHA1 revision id, and any tags that point to it. When unpacked, these strings are parsed to extract the most likely tag name. This allows tarballs generated by gitweb and GitHub’s “Download A Tarball” button to get useful version strings.
• Using “setup.py dist” to make a tarball, or “setup.py build” to copy code into build/, will compute the version and replace _version.py with a short form that just has the computed strings.

The result is that you get a useful fine-grained version string, updated every time you run your program, embedded into release products via the most common tarball-generation tools (setup.py sdist and git-archive). Developers will get detailed version information in their test logs (assuming you record __version__ in them, which you should), so other developers can reproduce their tree. Bug reports from end users will contain enough data (assuming they emit the version string) to reproduce their code, and to learn if they have local modifications or not. Release managers only need to run “git tag” when they decide to make a new release. And development/stabilization/release branches can be merged freely without worrying about what will happen to embedded version strings.

There are a few gotchas:

• the tool only handles git so far. I plan to add support for other systems in the future. Git is nice because git-describe is so fast. The –dirty flag does require that it stat every file (and possibly hash the contents), but that’s still pretty fast (25ms on my laptop for the jetpack tree). Other VCSes should be similarly fast, except for Darcs for which there is no concise version string except for specific tags.
• The git-archive expanded variables are abbreviated, and don’t distinguish between branch names and tag names, so sometimes we have to guess. Also you might have multiple tags pointing at the current revision (very common when your last release candidate gets promoted to final: the code sees both “1.3rc2″ and “1.3″ and picks the shorter one). It works pretty well when you build a tarball from a release tag, but building from something between-releases only gets you the full SHA1 (not the 1.4-8-gREVID form).
• if you wind up in a git checkout but either /usr/bin/git or .git is unavailable (maybe your git is named something else, or you’ve deleted the .git directory, or something), then the code will report a version of “unknown”. This happened on the Flightdeck (aka Add-On Builder) site, where it turned out they were getting the Jetpack code in a git submodule and then copying everything but the .git directory to a deployment server. This also happens if you’re bridging the code into a different VC system (e.g. using hg_git or git-svn or something) and using the result. This happens on the git-to-hg bridged version of the jetpack trunk, which is still used by some RelEng automation.
• “1.4-8-gREVID” is not PEP386-compliant: it can tolerate an “-rNNN” suffix (for SVN-style revision IDs), but not the more general “-gHEX” suffix that git-describe provides, nor the “-8″ revision counter. You may need to mangle the result if PEP386-compatibility of between-release version strings is important to you.

Jetpack uses Versioneer as of addon-sdk-1.4, with only a few stumbles so far. I’ve also switched python-ed25519 to use Versioneer. Let me know what you think!

But actually building them is a hassle, because the NaCl build process is so idiosyncratic. It consists of a 500-line undocumented shell script named “do”, and running it gets you 25 minutes of 100% CPU that executes in stony silence (all progress messages are redirected to a logfile). If you can wait that long, and think to explore the directory afterwards, you’ll be rewarded with a build/HOSTNAME/ directory that contains a libnacl.a and a set of header files that are pretty easy to use. What’s actually going on behind the scenes is that the script is exhaustively compiling and testing a large matrix of compiler flags (-O vs -O3 vs -O3 -funroll-loops), ABI variants, and alternative implementations. The goal is apparently to:

• select the fastest possible implementation and compiler options, using any assembly-language tricks specific to the processor (SSE3, etc)
• make sure the unit tests pass
• construct a performance report to send back to the authors

Unfortunately, this doesn’t play well with other build systems that might want to embed a copy, such as Python’s distutils, because:

• some compiler flags (-fPIC) are needed to build the .so files that python can load at runtime: distutils knows what these are, “do” doesn’t
• when building e.g. debian packages, the results will be used on other machines, so processor-specific optimizations aren’t ok (you might build your packages on a machine with some feature that’s not present on the machines that use those packages, so stick with least-common-denominator).
• running “do” requires a Bourne Shell interpreter, standard on unix systems but not so obvious on windows. distutils knows how to compile things on windows, but you have to tell it the source files, and it will run the compiler itself.
• having a separate “./do” compile step means that “setup.py build” is not enough, which means that easy_install won’t work, making it hard to use pynacl as a dependency in virtualenv or pip environments.

In addition, waiting 25 minutes for an otherwise small and elegant library to build is just a drag.

So I’ve pursued two projects this week. The first is to simply embed a mostly-unmodified (I did add -fPIC) copy of the latest nacl release into the pynacl source tree, and modify pynacl’s setup.py to use it. The build process becomes “cd nacl-*; ./do; cd ..; python setup.py build”. This is way easier than downloading+building an external copy and then pointing environment variables at the result. Takes just as long, but is easier to run. This work is in my “embed-nacl” branch (don’t be surprised if the branch is gone by the time you read this.. it may get merged to trunk). I also have a subset of “do”, named “dont”, which cuts out some of the unused pieces (like the C++ bindings and the large performance benchmarks), and runs in about 8 minutes.

The second is to embed a modified subset of the nacl sources in pynacl, making it look much more like a normal python extension module. This is in my “minimal” branch (again, this branch may get merged and be deleted).

This turned out to be pretty hard: the nacl “do” script synthesizes different versions of the same header file for each algorithm, and deletes many of them just after compilation. Also, the C code for e.g. crypto_hash_sha256 uses the same function name crypto_hash() as the code for crypto_hash_sha512, and depends upon #defines in the header file to prevent them from colliding. (the goal here is to let users call short function names and get the recommended algorithm, like crypto_box() instead of crypto_box_curve25519xsalsa20poly1305(), which is frequently a good idea, but not always).

So I had to modify the .c files to have non-colliding names, and write new header files to provide the recommended-algorithm mapping. I only copied in the portable implementations (leaving out the non-portable asm speedups). I also left out the multiple-ABI support, the try-different-compiler-flags speedups, the automatic performance tests, and the unit tests (although I hope to bring those back). But the result is something that compiles in 13 seconds, a 100x speedup, and which can be built with easy_install.

How does performance suffer without the specialized implementations and compiler flags? Here’s a comparison of the time it takes to call the various python functions (using 1000-byte messages, where applicable) in the two versions, on my 2010 MacBookPro (2.8GHz Core2Duo). The results are tolerable. The biggest victim is the Curve25519 scalar multiplication functions, since the whole algorithm was specifically designed to take advantage of 64-bit operations not available to the generic version (which needs to run on 32-bit machines). This slowdown also hits the high-level box()/box_open() functions, bringing them up to 5ms on my laptop. crypto_sign uses an early version of Ed25519, and nacl does not yet have an optimized version, so it takes 6ms in both branches. The next version of nacl will include the same carefully optimized version of Ed25519 as the SUPERCOP benchmark suite, in which signing takes about 19us, so this will also become a noticeable slowdown.

But I think the result is still pretty good: for many applications where you’re using Python, 5ms is just as good as 200us. And this branch makes it much easier to depend upon pynacl than the others.

My next steps are to clean up the branch a bit, talk to k3d3 about which approach seems the best, get it merged upstream, and then hopefully get it copied to PyPI. And then investigate how hard it’d be to build the alternative implementations: maybe with some distutils hacking, we could build all the variants (and see which ones actually work), do some quick performance tests, then call setup() again “for real” with just the fastest ones. That should shave off most of the slowdown (leaving the compiler-flag gains), but still give us something that compiles easily.

There are several different implementations of the Ed25519 signature system, and they each use slightly different key formats. While writing python-ed25519, I wanted to validate it against the upstream known-answer-tests, so I had to figure out how to convert those keys into a format that my code could use.

The best reference is the original paper, which explains everything you need to know to implement the basic algorithm. I currently know of three or four implementations (many of which are on the Ed25519 software page):

• a portable C “reference” version
• a 10x faster AMD64-optimized C/assembly version (in two flavors, with different speed/memory tradeoffs)
• an educational (but unusably slow) pure-python module
• my Python binding to the C reference version

djb’s NaCl library (as of version 20110221) includes an older copy prototype of the portable C version, in nacl-20110221/crypto_sign/edwards5519sha512batch/ref/ (as far as I can tell, it doesn’t implement quite the same algorithm: it doesn’t include the pubkey in the hash when computing S. UPDATE: djb told me that nacl-20110221′s code is just a prototype, and will be replaced with the real Ed25519 in the next release). The SUPERCOP benchmark suite includes both the portable C and the two AMD64-specific flavors (in supercop-20110704/crypto_sign/ed25519/, under the ref, amd64-51-30k, and amd64-64-24k subdirectories). The pedagogical pure-python implementation is referenced by the ed25519 web page, but since it takes tens of seconds to compute each signature, it’s really only useful for validating the output of other implementations.

My python-ed25519 library provides Python bindings to the portable C version, using code taken from the SUPERCOP suite.

These different implementations all arrange the keys and signatures slightly differently. NaCl’s goal is to provide safe high-level box/unbox functions, so the only interface it offers will copy the entire message into the output and append a signature to it (not so convenient for building into other protocols). SUPERCOP’s goal is to measure the speed of each operation, so its interface isn’t so convenient for general use either. I’ve tried to give python-ed25519 a good interface for use in other projects.

To remind myself about how these different implementations handle keys, here’s an overview.

Ed25519 keys start life as a 32-byte (256-bit) uniformly random binary seed (e.g. the output of SHA256 on some random input). The seed is then hashed using SHA512, which gets you 64 bytes (512 bits), which is then split into a “left half” (the first 32 bytes) and a “right half”. The left half is massaged into a curve25519 private scalar “a” by setting and clearing a few high/low-order bits. The pubkey is generated by multiplying this secret scalar by “B” (the generator), which yields a 32-byte/256-bit group element “A”.

When signatures are made, two values result: R and S (both 32-bytes, so the overall signature is 64 bytes long). R depends upon the right half of the expanded seed and on the message. (In traditional DSA, R is randomly generated, and the security of the private key depends upon the quality of that randomness, leading to some high-profile failures). S is the real meat of the signature, and is a function of everything (including the pubkey).

Private signing keys can be generated from just the 32-byte seed, but require additional work before you can make signatures (SHA512-based expansion, the bit setting/clearing massage step, and pubkey exponentiation). Likewise public verifying keys can be derived from either the private seed or the private exponent. (note that it’s the multiplications that take the most time: everything else is trivial by comparison).

If you care more about the speed of operations than storage space, you’d want to store the expanded versions. Or, you might want to store as little information as possible, and accept the performance penalty of re-deriving things when necessary. Different implementations choose different tradeoffs.

With that background, here’s what the different implementations do:

• slow pure-python “ed25519.py” implementation, with Known-Answer-Tests in sign.py and sign.input:
  • publickey(sk) takes the seed, returns the 32-byte pubkey
  • signature() takes the message to be signed, the seed, and the pubkey, returns the 64-byte signature
  • checkvalid() takes the signature, the message, and the 32-byte pubkey
  • each line of the test data contains four fields (joined by colons):
  • seed+pubkey (64 bytes)
  • just the pubkey (32 bytes)
  • the message (variable length)
  • the signature concatenated with the message (R+S+msg)
• NaCl-20110221 implementation:
  • keypair() returns two values. The first (signing key) is the private scalar (32 bytes) concatenated with the “right half” (also 32 bytes). The second is the pubkey (32 bytes)
  • sign() takes the signing key and message. The NaCl version doesn’t take the pubkey, and thus diverges from the ed25519.py implementation: S=H(r+msg), not H(r+pubkey+msg). ed25519.py matches the paper, as does SUPERCOP, but NaCl does not. sign() returns R+msg+S
• SUPERCOP-20110704 implementation:
  • keypair() returns two values. The first (signing key) is the seed (32 bytes) concatenated with the generated pubkey (32 bytes). The second is the pubkey (32 bytes).
  • sign() returns R+S+msg
• python-ed25519 (using the SUPERCOP code but changing the output):
  • ed25519.create_keypair() returns a SigningKey object and a VerifyingKey object
  • SigningKeys can be serialized with to_bytes() (which returns the same 64 bytes as the SUPERCOP code that it wraps: the seed concatenated with the pubkey A), or with to_seed() (which returns just the 32-byte seed).
  • VerifyingKeys can be serialized with to_bytes(), which returns the
    32-byte pubkey A.
  • sk.sign() returns a 64-byte signature (the concatenation of R and S) without returning a copy of the message too. (internally, this invokes SUPERCOP’s crypto_sign() function with an output buffer large enough to hold R+S+msg, then discards the message).

This is summarized in the following diagram:

I’ve published some MIT-licensed Python bindings to djb++’s portable C implementation of this signature scheme. They’re available here:

• https://github.com/warner/python-ed25519
• or easy_install ed25519

Some highlights:

• signing keys and verifing keys are both just 32 bytes
• signatures are 64 bytes
• key generation and signing each take about 2ms on my 2010 MacBookPro
• signature verification takes about 6ms
• 128-bit security level, comparable to AES-128, SHA256, and 3072-bit RSA
• No entropy needed during signing (signatures are deterministic)

There are amd64-specific assembly versions that run even faster, in just a few hundred microseconds, and for bulk operations you can do batch verification faster than one-at-a-time verification. So you can perform thousands of operations per second with this algorithm (and hundreds with this particular implementation).

It’s very exciting to finally have short+fast signatures (and also, through Curve25519, key-agreement and encryption): it opens up a lot of new possibilities. When public-key encryption was first invented, keys took so long to generate that folks assumed that each human would have just one: all sorts of mental baggage was built up around this restriction (ideas like never sharing signing keys, keys representing people, and the need to distribute keys separately from fingerprints). When you can easily generate a new key for each message or object or operation, we can let go of some of those psychological fetters and build something new.

(Note that “Curve25519” uses the same basic curve equation, but only provides Diffie-Hellman key agreement [and, by extension, public-key encryption]. It can’t be used to create signatures that can be verified by third parties: for that you need Ed25519. A portable Curve25519 implementation can be found in curve25519-donna, and includes a Python binding that I wrote too)

Take a look and let me know what you think!

For a while now, I’ve been noodling around with a cryptographically-secure "invitation protocol" of sorts. I want something that can start with Alice sending an Invitation Message to Bob, and end with fully secure bidirectional communication.

The use cases I have so far are:

• secure file-transfer tools, as a browser plugin or standalone application. The program would have an address-book, each entry would have an "Inbox" for that person, you would send them a file by drag-and-dropping it onto the Inbox. This invitation protocol would be used to populate the address book.
• secure communication tools, like Petmail or an encrypted email/IM client, in which you want to add address-book entries so you can later exchange messages with the right correspondent.
• Tahoe grid membership tools: you would use an Invitation to securely add another person’s client/server node to your grid, maintaining enough information to perform Accounting and server-selection properly.

but there are lots of others. Most applications with any sort of social component need a way to add "buddies" or correspondents. There are plenty of centralized systems (like IM networks, where the namespace is controlled by a single company), and a handful of insecure decentralized systems (like email addresses). I want this protocol for secure decentralized systems, in which the primary identifier for each participant is a public key of some sort.

Read on if you’re interested in cryptographic protocols..

A big usability-barrier for such systems is the bootstrapping problem: getting from a casual (insecure) everyday channel to the persistent secure long-term channel. A lot of tools punt on this and assume the keys are distributed ahead of time, or managed by some administrator. I’m interested in systems for which the only mode is "secure" mode, and which are as easy to use as traditional insecure systems. The usability standard is that the humans do not need to do anything more than they’re already doing: the system would look no simpler if you ripped out the security, so we might as well leave it in.

Imagine you’re chatting with someone at a conference, or at lunch, or in an IRC channel, and you want to send them a file, or pick up the conversation later, or whatever. You want your user agent to learn about their user agent, so that you can use it to send them a message later on. I’d like this to be properly secure, so every message is signed and encrypted, which means the agents need to learn each other’s keys. What can the two humans exchange that will let their agents establish that cryptographic connection?

The usual (worst case) pattern, with something like PGP, is that both people must pre-arrange for some common server to hold copies of their public keys, indexed by an email address, and to have secure hashes of those keys on something transferrable in their pocket (frequently printed on business cards). They can exchange business cards and type in their contents later (downloading the keys, computing the hashes, and comparing against the cards). Or, if they have an online connection already, they can paste large key files to each other, and perhaps paste+compare fingerprints. But we have plenty of experience showing that these schemes aren’t sufficiently usable, and needing to type in all that material discourages casual real-time use.

In the protocol I have in mind, one of these people sends an "Invitation" to the other, and the second person accepts it. Alice, the inviter, asks her computer for the invitation object (a file or a short string), and she somehow hands it to Bob: maybe she emails it to him, or pastes it to him over an IM channel, or recites it to him in person or over the phone. Bob drags or pastes or types this invitation into his computer. Then the two computers talk to each other, exchange keys, and at the end of the process, each computer has an address-book entry for the other. Later, their applications can use these keys to perform secure message exchanges with each other.

The ground rules for this protocol (including some assumptions and basic engineering tradeoffs) are:

• Alice has a "secure-enough" channel to Bob already. "secure-enough" means we’re comfortable pretending that whatever she emails or pastes or speaks to him will remain secret and intact (not modified by an attacker) long enough for Bob to use it. Alice and Bob can take extra steps to make it more secure, if they want.
• The channel isn’t long-term secure: we assume that the contents will become public eventually, so we aren’t allowed to put any long-term secrets in it
• The channel is one-way and one-shot: if Alice sends her invitation via email, the protocol isn’t allowed to use email to get the response back. (this would impose burdens on the users, and require more complex interaction between the protocol implementation and existing tools)
• The Invitation is thus single-use and only valid for a limited time, maybe 24 hours. We’re comfortable with a "Trust On First Use" policy: an attacker who both learns and acts upon the invitation early can impersonate Bob. But there will be a way to do an out-of-band comparison later, to completely rule out a man-in-the-middle.
• Since the Internet is no longer a network of computers that can all talk to each other, we assume that either agent may be behind a NAT box. We allow applications to bake-in the address a publically-reachable "Relay Service", which helps them exchange messages for the duration of the protocol. We don’t allow this service to affect the security of the system: it could mount a DoS attack, but otherwise is no more powerful than any other MitM.
• The goal of the protocol is to securely get Alice’s address-book entry into Bob’s address-book, under a petname for "Alice" that Bob picks when he accepts the invitation: we call this entry Ealice. We also need symmetry: to get Bob’s entry into Alice’s address-book under a petname that Alice picks when she sends the invitation, which we call Ebob. Part of these entries is the address of a relay-service or other helper to exchange subsequent application-specific messages.
• All these address-book entries are public knowledge: no need to keep them confidential during the protocol (this turns out to affect the protocol in significant ways)
• We have a secure/fast/short public-key signature scheme: Ed25519. We want a 128-bit security level.

I’ll post more later about the protocol I’m building to satisfy these constraints. Let me know what you think!

I’ll be leading a session on using modules. See you there!

I’ve been working at Mozilla for about a year and a half. Before Mozilla I was at a startup (now gone) named AllMyData.com, which ran an online backup service aimed at home users and small businesses. For the backend, we developed an open-source encrypted distributed filesystem named Tahoe-LAFS (still thriving), with the relatively-unique feature of “provider-independent security”: the servers hold ciphertext, and only your client (and anyone you choose to share them with) holds the decryption key. By relying upon the servers for availability but not security, you can use servers that you wouldn’t trust otherwise. The system is designed with “POLA” in mind, the Principle Of Least Authority, which improves safety by giving each component as little power as possible; just enough to do its one job.

Mozilla’s Sync system, which helps keep your bookmarks and other browser data synchronized between multiple browsers, uses the same principle. The data is encrypted before it leaves your computer, and our servers don’t get to see the key. The similarity in philosophy between Sync and Tahoe was a big part of the reason I joined Mozilla: I had found a bunch of like-minded people with whom I could promote these ideas, with a wider audience (400 million users and counting!) than anything our tiny startup could reach. Shortly after joining, I helped build the credential-exchange protocol that Sync uses to add a new computer to your account (the “copy this code” J-PAKE-based dialog panel).

But most of what I’ve been doing for the last 18 months is Jetpack, now better known as the Add-On SDK. It’s a new way to tackle the admittedly painful process of writing add-ons for Firefox (and eventually other XUL-based programs, like Thunderbird).

Add-ons in Firefox have always been a bit.. organic. There’s an awful lot of internal interfaces, and add-ons authors have had to dig through a lot of that to find a useful point to attach or override some code. It’s messy, and fragile: it’s difficult for the Firefox developers to keep those internals stable while making large-scale improvements to the browser, so add-on authors are fighting an uphill battle to maintain compatibility with newer releases. With our new faster release cadence (every couple months!), the problem becomes even more challenging.

Jetpack was intended to make the simple things simple. It has a library of interface modules for doing common add-on tasks, like creating a UI, and modifying web pages. We, the SDK authors, will keep updating these modules to match the Firefox internals, so add-on authors won’t have to. One of the promises of Jetpack is that the high-level add-on code doesn’t need to change much at all to keep up with newer browsers.

The SDK has two parts: this library of modules, and a program that assembles your code with the right set of modules and produces the XPI file, ready for installation or upload to AMO. If you think about it in terms of gcc and other compilers, this program is a linker: it figures out which modules are required and combines them into an executable bundle.

The linker is written in Python, and since that’s been my dominant language for the last ten years, most of my Jetpack work has been on the linker side. I’ve also been heavily involved in developing the security model, to make Jetpack-based addons safer in the face of bugs and compromises. I’ll be explaining a lot more about the security model in this blog over the coming months.

And no description of Jetpack is complete without also mentioning Flightdeck, better known as the Add-On Builder. This is an online IDE for writing add-ons. On the backend, it runs a copy of the SDK to generate the XPIs. It allows folks to get into the add-on writing business without downloading or installing an SDK: just start writing the code! A big goal of Jetpack is to make add-on writing accessible to a wider developer community, leveraging Javascript skills that web-devs already have, and an online tool like Builder removes a lot of the unnecessary hurdles for that audience.

Welcome!
-Brian