Better Know a WebDev: Kumar McMillan (aka kumar303)

Welcome back! No, we didn’t forget about our blog profiles, but your humble chronicler was away and neglected to get a substitute.

So we return from our brief hiatus with the powerhouse engineer, Kumar McMillan. In his own words…

Kumar in his element

What do you do at Mozilla?

I work on addons.mozilla.org, a large Django app that hosts Firefox add-ons. We just launched marketplace.mozilla.org for apps which is an exciting new take on what an app store can be; it runs on the same codebase as the add-ons site. Open web apps can run on any device that supports modern web standards and WebRT, our web runtime. If you buy an app through Mozilla’s marketplace you can use it in any environment that we support (currently: Mac OS X, Windows, Android). If we don’t yet support it, you can roll your own by hooking into our receipt verification service. Open web apps offer a compelling promise: write your “native feeling” app once using web standards and deploy everywhere. Ours is the first app store that doesn’t lock you into a device and a vertical development silo. It’s currently not that straight forward to use HTML5, CSS3, and JS to make an app feel “native” but the incentives are pretty huge for the community to improve on this so I think we’ll see it happen (e.g. Twitter’s bootstrap, Application Cache, etc). There are some other interesting parts of the marketplace that I’m working on specifically like in-app payments.

Any fun side projects you’re working on?

I spend a lot of my free time working on a volunteer-run Internet radio station in Chicago called CHIRP Radio. I head up their Tech Dept. and help build apps for listening to CHIRP and tools for DJs to use while operating the station. CHIRP Radio is non profit and, it’s funny, there are a lot of similarities with Mozilla albeit on a much smaller scale. We develop everything in the open and engage with the open source community, for example.

I also host a radio show Tuesday mornings as DJ Bylamplight and that’s something else I spend a lot of time on: music. I like to balance my cerebral world of hacking on code with music. It’s emotional. I love to DJ songs for people because it affects their mood. I also dabble in a recording studio that I share with several engineers and bands. I just finished producing an album for Summer Girlfriends which was a lot of fun. Oh yeah, and I help run a vinyl label called Addenda and a cassette label called Plustapes. People love cassette tapes. It is true.

How did you get started in web development or programming?

I went to The School of the Art Institute of Chicago because they had a Sound Department which I thought was a neat idea. As part of my degree I did a lot of different things though: Film, Performance, Art & Technology. In the latter, I took a class where the instructor bravely had us write C code to process images (even though no one had any programming experience). We literally just made X and Y loops of pixels and wrote programs to distort images. It was super fun and from that moment I was hooked. However, art school made me realize I didn’t want to be in the art scene. The Web was exploding at the time and it felt like the place to be so I taught myself web programming (initially in PHP). The creative process of writing code isn’t very different than that of building sculptures or painting. There’s just more typing involved.

How did you get involved with Mozilla?

I’ve used the same online bookmark system since 2002 (don’t ask) and long before I joined Mozilla I had a folder of bookmarks called Mozilla. I looked up the first thing I ever bookmarked in that folder and it was this link back in 2005: http://www.mozilla.org/quality/smoketests (gone now), which was about the extensive automated test suite for Firefox. I was totally fascinated by this test suite and how they had heat map regression tests for when CSS implementations broke visual layouts. I was very inspired by this when I started to do automated testing. Mozilla felt like a pioneer in this area and many other areas.

Several years after that, Mozilla started doing a lot of work in Python and several people I knew from the open source community started working there. I liked what they were doing with Python and since they were already using some open source libraries I was working on, it was a natural fit. Actually, most of the people we hire in webdev are authors of libraries I’ve used or are authors of blog posts that turn up regularly when I search for tech help.

What’s a funny fail story or mistake you can share?

I once deployed a bad change to addon.mozilla.org‘s web service used by Firefox clients that triggered an unexpected traceback on every request. Since these service URLs serve about 8,000 requests a second it killed the cache and basically started to melt down the server cluster (the graphs were ominous). Luckily my new code was behind a feature flag so we could turn it off immediately without having to re-deploy. There ya go kids, always use feature flags. It was indeed a reminder that our flag based strategy was a good one.

What’s something you’re particularly proud of?

It still needs some work but I wrote JS TestNet to allow addons.mozilla.org to run small integration tests for our JavaScript functionality. support.mozilla.org is using it now as well. Providing a continuous integration framework for JS testing only solves half of the problem though. There is a huge gap in the JS web dev community where barely anyone writes automated tests. All the popular patterns around jQuery and jQuery plugins do not encourage you to write testable code and it’s hard to retro fit tests into code like that.

What’s coming up that you’re excited about?

I’m excited about the Mozilla apps project and how our payment systems might instigate new online business models. I’m excited about apps in general. I can’t wait to build apps for cars. I love my iPad, my Android tablet, and my Android phone. If we stand by and do nothing, the web is going to become a second class citizen in these apps-centric devices. Mozilla can change that. I think we might even get some major players like Facebook or Google to join in our crusade to appify the the web in an open and egalitarian way if we prove that our concepts will work.

I’m also excited about Sagrada and Petri, two projects that aim to build a scalable platform for hosting web applications. One day Mozilla may be able host community apps on these platforms and further our mission to keep the web open and accessible to all.

What question do you wish you’d been asked?

How do today’s open source communities relate to ancient hunter / gatherer tribes where shared resources and equality were key components?

What do you listen to while you work/program?

I listen to CHIRP Radio, WBEZ, KCRWWFMUThis Is My Jam, Soundcloud, my record collection, and my mp3 collection. Thanks for asking.

Better Know a WebDev: Greg Koberger aka gkoberger

Welcome to another thrilling installment of your favorite, and only, recurring series on this blog: Better Know a WebDev!

This is a special week, as we’re featuring someone who just moved from web development into a product role with our Add-ons team. Give a big welcome and congratulations to Greg Koberger!

What do you do at Mozilla?

For the past year and a half, I’ve been a web developer working on the Mozilla Add-ons site. I wrote both front- and back-end code, and every once in a while I managed to sneak in a little bit of design work. I recently traded in vim for Google Docs, however, and made the switch to a more product-centric role. I still get to work with the same people on the same projects, however I’ll be writing specs and designing flows rather than writing code.

Any fun side projects you’re working on?

One of most recent (and relevant) mini side projects is a Firefox Quine. It’s not a true quine, since it loads some external assets (like jQuery). However, the concept is the same: if you were to copy and paste the code that makes up the logo, it would output itself. We’re currently getting the design printed on t-shirts.

How did you get started in web development or programming?

I was lucky—my parents always made sure I had a good computer. I started by tinkering with HyperCard, and eventually decided to try making websites. I still remember how asking my parents for my first hosting account went: “Why do you need other people to see it? Can’t you just play with it yourself on your computer?” Luckily they finally agreed, and I started teaching myself PHP and JavaScript.

How did you get involved with Mozilla?

While in college, there was an alumni talk about working at Mozilla. I missed the talk, however I emailed the guy after since I was curious. He encouraged me to apply, and I flew out to California to interview. I fell in love with the company. The people I interviewed with were awesome, and I was amazed by the company culture. Everyone just wanted to make the web a better place.

What’s a funny fail story or mistake you can share?

Back when I first started at Mozilla, I managed to delete all my private keys. So, some of the other webdevs set up a tissue-box-based “Computer School Fund” for me. I made $20 for deleting my private keys, so I like to think I got the last laugh.

What’s something you’re particularly proud of?

Back in November, awareness of SOPA was just starting to pick up. The day before the initial hearing, a bunch of us decided to try to raise awareness. It was incredibly last minute, but we knew we had to do something. We could get the message out via the snippet on about:home, which serves about 300MM requests a day. However, we had nowhere to link to. SOPA was new, and there wasn’t a definitive source with a few easy-to-digest bullet points. Plus, we couldn’t really send millions of hits to a website without giving them fair notice. So, we made our own. I sketched it out on paper and wrote out the copy. Then, as I designed it in Photoshop, my fellow webdev Potch sat next to me and wrote the code. We didn’t even have time to share files; he just watched what I was doing and wrote the code. It took us about an hour—which is all we had, since it still had to be deployed and tested. We couldn’t have gotten it out without help from all the different teams at Mozilla—everyone from legal to QA to IT to webdev worked late to make it happen. It was a huge success. We got the word about SOPA out to millions of Firefox users within 24 hours. We were able to use the same site a few months later for the SOPA blackout.

What’s coming up that you’re excited about?

I’m really excited about my new product management role. I love new challenges. It’s a great time to start this job, since a lot is going on with AMO right now. “Marketplace” (apps and add-ons combined, with the ability to charge) is going to be a huge project in the upcoming months. I can’t wait to see what our team can make in the upcoming months.

What software do you use day-to-day for the work you do?

My main machine is a 13″ Macbook Air running OS X Lion. I use vim for writing code, which all ends up on GitHub. I rely heavily on Firefox and Firebug for testing and debugging. I spend a good amount of time in Photoshop, mocking things up before I start coding. I couldn’t survive without either Rdio or Pandora playing in the background.

Mozilla WebDev: April 2011

Better Know a WebDev: Will Kahn-Greene aka willkg

Welcome back to the third epic installment of Better Know a WebDev!

We started with our illustrious director, Mike Morgan, and then someone volunteered that the chronicler, yours truly, should be up next. But now we’re just going to randomly hop around through our entire web dev family. And there’s a lot of us!

Up this week, put your internet hands together for Will Kahn-Greene!

WillKG

What do you do at Mozilla?

I’m a software engineer in the webdev group that works on the software that runs support.mozilla.org. That entails working directly on kitsune—the software that runs support.mozilla.org—and its dependencies. I also spend some time working on making development life easier.

Any fun side projects you’re working on?

I have too many side projects, so I sort of bounce between them month-to-month. I’m the current maintainer of Pyblosxom—a file-based weblog system. I’m a cofounder of MediaGoblin and help maintain project infrastructure. I run Python Miro Community. I also work on Miro and Miro Community, though in a diminished capacity since I left PCF to work at Mozilla.

How did you get started in web development?

Long story short, I started writing web pages and CGI programs in C, then Perl, in undergrad. I learned enough to become an “authority” and taught a class on web-design and authoring in Microsoft Online Institute. The version of MOLI that I used was a web-based chat system. I thought it was pretty lacking, so I did my senior undergrad thesis studying web-based instruction and building a better system. From there, I did a bunch of consulting on large web-sites using IIS/ASP then Java EJBs/JSP/Servlets. After consulting, I switched sides (producer → consumer of web apps) and worked for a company writing a scraper for financial data.

How did you get involved with Mozilla?

In undergrad, we had Mosaic installed on the computers in the lab and I used that until I switched to Netscape. I used that for a few versions, then got really excited when Mozilla formed and started doing M releases. I’ve been using a Mozilla browser ever since.

I worked for Participatory Culture Foundation on the Miro media player, but while there worked on a variety of side projects including Firefox extensions, scripts that augmented the PCF Bugzilla instance, Universal Subtitles, and also I wrote the code that showed enclosure information in the feed preview page for Firefox 3 (or 3.5—I forget when it landed).

When a Mozilla recruiter asked if I’d be interested in working for Mozilla as an employee, I jumped at the chance.

What’s a funny fail story or mistake you can share?

Seems like everyone’s done the “rm -rf /” thing. Me, too. Though it was less funny and more of that “oh, what have I done!” sinking feeling of despair.

The fail story I often tell people who seem apprehensive about computers and doing things with them happened when I was in undergrad. I had been programming since like 4th grade in a bunch of environments and knew my way around PC computers, Windows… but I hadn’t had much experience with Macs. In college, we had a Mac lab. So I trot into the Mac lab to work on my first homework assignment for CS 1 and I sit down and I look at the Mac and it had no visible on/off switch. I looked on the sides and the back. Nothing—no switches or anything on/off-related. I looked for instructions nearby. Nothing. I was so embarrassed that I couldn’t figure out how to turn the computer on that I left and got a 0 on that homework. Turns out that the “on” key was on the keyboard in the upper right hand corner and had a triangle on it. For this particular Mac, there was no “off” switch. I never would have guessed that. The moral of the story being that computers are mysterious and a total pain in the ass even for people who have lots of experience with them.

What’s coming up that you’re excited about?

I started at Mozilla in September 2011, so I’ve only been here a short while and it’s hard not to be excited about everything. Seems like every day I hang out with the Mozilla folks, I learn something new and about the existence of ten things I wasn’t previously aware of.

A less hand-wavey answer might go along these lines: I’m excited about the momentum towards open web apps and reducing the barriers to web development. Literacy with web development is a powerful thing. Being able to run apps everywhere is also a powerful thing.

Today! Mozilla Webdev “Ask Me Anything” (AMA) on Reddit!

As promised, today, a bunch of Mozilla Webdevs are on reddit to answer your questions! For more information, check out the announcement.

Click here to get to the Mozilla Webdev “Ask Me Anything” on Reddit.

We will be there from 10am Central European Time to the evening, Pacific time, so in total about 18 hours. We hope this way people in most time zones will get the chance to get their questions answered.

Come by and ask us all you never dared to ask but always wanted to know!

Better Know a WebDev: James Socol

Welcome back to our Better Know a WebDev series! This week, we’re looking at, well, me!

What do you do at Mozilla?

I’m the Community Platforms Manager. I’m here to help the people on the Support, MDN, and Community Tools engineering teams turn out great products and grow as engineers. Since I get to work with a number of different projects, I have the opportunity to look for ways these applications can work together and create a better experience for our users and better data for our engineers.

Any fun side projects you’re working on?

Lately I’ve been working with Django and MongoDB for fun, and trying to help out in that part of the Django ecosystem where I can. I also maintain a few libraries, like Bleach, and write useful—or useless—IRC bots from time to time.

James' favorite topic.

As a proud resident of New York City, I’ve been spending a lot of time trying to push Mozilla further into the New York tech scene, mostly by giving talks on things like continuous deployment, building really large scale web apps, and even why Open Source Software is so great.

How did you get started in web development?

A friend of mine showed me his Geocities page, back when you still had neighbors on Geocities. I jumped in, found a tutorial and went from there. In college, I got a job doing some app development and maintenance that exposed me to a bunch of languages and working with a team, where I quickly learned the phrase “best practices.”

How did you get involved with Mozilla?

I started using Firefox when it was still Phoenix, and being a user was about the limit of my involvement until 2009, when I applied for a job here.

One of the things that initially surprised me was that all of our websites are Open Source, but then I realized there was no other way for web development at Mozilla to operate. In the past couple years, I think we’ve improved a lot here by being more outgoing members of the Django and Python communities, and we have some regular contributors now.

What’s a funny fail or mistake you’re willing to share?

Well, there was that one time I ran an UPDATE statement without a WHERE clause on a production database. I think we all have one like that.

At Mozilla, the most epic was probably the SUMO 2.0 release, which took a solid 12 hours, took down a cluster serving over a dozens sites, and—at around 4am, a few hours after we thought we were in the clear and had all gone to bed—fell over, requiring a roll back and for us to do it again a week later.

I learned a lot in that week. And now whenever someone in our IT group calls me, the ringtone sounds pretty urgent.

What’s one thing coming up that you’re excited about? What makes it exciting?

2012! Community Platforms is a new group, and Community Tools in particular is a new team, and I am really looking forward to seeing what we accomplish in 2012. Growing the Support and MDN teams and building the Tools team are going to be fun challenges. One of my goals is to start turning our siloed applications into platforms to empower our users and developers, and I can’t wait to see what new, innovative ideas our community builds on them.

Timing AMO user experience

There are lots of ways to measure the performance of a site. On addons.mozilla.org we measure a few of them: how long it takes to render pages, the cache performance and responsiveness for content delivery networks globally. But with the advent of the navigation timing API in Firefox 7 we’ve been able to add the most important measurement of all – the actual performance in the browser.

Along with a few other Mozilla sites, we’ve been using pystatsd and Graphite for a while. This produces useful graphs of the site health and performance. For example, this graph plots HTTP responses we serve out from Python (this is not the amount of traffic to our site which is much larger).

To facilitate using the timing API we wrote a module between statsd and Django called django-statsd that, amongst other things, provides a way to interface between the browser and Python backend. The library has support for boomerang, a great front end timing library. I quickly wrote a much simpler service called stick, which sends a small subset of timings that we want.

Using stick is straightforward – once you’ve setup your Python backed and included your JavaScript, you call it in your page. For example:

stick.send('https://to.your.site/record')

The API returns with timestamps for each of the events. To give us meaningful numbers, we then subtract the timestamp for each of the events from navigationStart so we can tell the timings relative to that start point. And here’s the end result in our Graphite server:

For internal sites we collect 100% of the data. However, on our live site we get quite a lot of traffic, so we’ve rolled this out by collecting only 1% of traffic. We’ll start increasing this percentage once we are confident this won’t be causing problems. We’ll also be adding it to the about:addons page which gets quite a bit of traffic.

We’ll be tweaking things to ensure we pass through the right numbers and interpret them correctly. The amount of data is probably too low to be useful, right now, but I can’t help but wonder what the peak around 12.30pm today was…

Better Know a WebDev: Mike Morgan (a.k.a. Morgamic)

Our group has grown a lot over the past few years, as we’ve taken on new projects and done a whole lot of amazing new things. Naturally, as we’ve grown we’ve become a group of more diverse backgrounds, skills and interests—everything from awesome JavaScript developers to our growing Hadoop expertise.

So this is the first post in a new series, Better Know a WebDev, where we’ll introduce everyone, and hopefully get to know each other a little better!

We’ll start with the big man, the head honcho, Mike Morgan, a.k.a. morgamic, one of the original Mozilla web developers and now Director of Web Development.

Morgamic and Lars

What do you do at Mozilla?

I’m the Director of Web Development. My job is to help people in webdev achieve great things and grow as engineers—and hopefully as people. I spend a lot of time on recruiting, but one of my unwritten jobs is serving as the “glue” between strategy and operations. I help everyone understand why projects are important and how strategic goals break down into day-to-day tasks. Mostly, I try to maintain a great culture for our team.

Any fun side projects you’re working on?

I spend most of my free time playing basketball or dabbling in photography. I’m pretty notorious for taking way too many fisheye photos.

How did you get started in web development?

In March of 2001 I taught myself HTML, CSS and PHP because I was bored during Spring Break and didn’t have any money to go anywhere. In 5 days I created a blog, picked up HTML and CSS. My first blog didn’t have tables, which was pretty frustrating because I quickly discovered all the crazy bugs with IE 5.5 and IE 5.1 (Mac). From there I started working part time on websites and just fell into web development.

How did you get involved with Mozilla?

My first Mozilla project was redoing bugzilla.org, which is why I’m still on the developers page. The bulk of my work came later in 2004 and 2005 when I got involved with addons.mozilla.org, Bouncer (download.mozilla.org) and aus2.mozilla.org.

One of my most exciting times at Mozilla was the Firefox 1.0 launch. I wrote a blog post about it back in 2004. From there I just kept working on web services and sites for Firefox, going on to build and scale a lot of the original Firefox services. Since I was pretty busy, I usually looked like this:

morgamic, 2006

What’s one mistake or funny fail story you’re willing to share?

Honestly, I can’t share the best fail stories publicly. But I’d say that accidentally wiping a production database with a bad DELETE statement is probably my biggest screw up. Don’t do that. Ever. There is no excuse and it’s embarrassing. Though if you do it once I hope you’re smart enough to never do it again. Just do it with a stage database.

What’s one thing coming up that you’re excited about? What’s exciting about it?

I want to play with camera and sound APIs in JavaScript. To have this all freed from Flash is a liberating and exciting concept. We are stepping into a time in our history where the web can literally do anything we can imagine. It’s so damn exciting—it’s just a wonderful time to be alive.

Mozilla is taking on proprietary app stores, identity and the mobile market. What a year 2012 will be. Tackling apps—freeing so much great content and capabilities currently locked down by one or two companies—it’s very exciting. When Mozilla gets this right, and I believe we will, it will be a great thing for everyone who uses the web on a desktop, phone, tablet, car, plane, refrigerator, etc.

What’s one question you’d like to be asked but haven’t been?

I guess I’d ask what I was most proud of, or what motivates me. Honestly, it’s pretty simple: my teammates. I’m surrounded by brilliant people who teach me something just about every day. The favorite part of my job is seeing them reach their goals and knowing I had a small part in it.

Webdev 2011

Coming up: Webdev “Ask Me Anything” (AMA) on reddit

Update, Jan 20: The Reddit AMA is going on today! Click here to join us and ask questions!


As web developers at Mozilla, we take pride in making web applications that are used by millions of people every day. We work hard to promote the Open Web, both by showing off what the latest in web technology can do and by actively developing our projects in the open: as open-source projects on github.

Whenever we go to meetups, conferences, or chat with community members, people ask all sorts of questions, like:

  • How do you make sure your websites are snappy for users across the globe?
  • I found a bug in one of your websites and would like to help fix it! How do I get started?
  • What frameworks do you use? And how many servers do you run certain projects on?
  • Why are you all so good-looking?

To open the conversation to a larger community, we’ve decided to run an “IAmA” on Reddit, on Wednesday, January 18, 2012 Friday, January 20, 2012.

We’ll open a thread there in the morning, Central European Time, and will have plenty of people on the Webdev team participating until the evening, Pacific Time, so we should be able to touch most time zones one way or another.

Whether you are a seasoned Mozillian or a “newbie” in the community, whether you are a passionate Firefox advocate or pragmatic browser user with an inquiring mind, step by and talk to us! We’d love to hear from you.

We are Web Developers on the Mozilla Webdev team: Ask Us Anything.

For a link to the actual “AMA” post on reddit, visit this blog on January 18 January 20 or follow us on Twitter. If you have any questions, feel free to comment here or visit us in #webdev on irc.mozilla.org.

Bringing Firefox Alive with Popcorn.js

For most people who work with computers for a living, holidays are a time for family… and providing endless tech support. As I was huddled over a half working computer, someone asked what an add-on was. I spend most of my time working with and on add-ons — so the concept seems obvious to me. As I found out this past Christmas, it’s not to everyone. My otherwise-fairly-computer-literate family had a hard time understanding where the browser ended and websites such as Facebook or Google started — so explaining the concept of add-ons was not nearly as easy as I thought it would be.

While over 85% of people using Firefox have at least one add-on installed, most people still don’t really know what an add-on is or the possibilities they provide. To fix this, the add-ons group teamed up with Mozilla Creative and an outside firm, Poke, to create a short video explaining what add-ons are and what they can do.

After seeing the video, we knew we had to do something awesome with Popcorn.js.

Watch The Video
On the page, click “Learn More” to play the video.

I recommend watching it in Firefox, since we interact with the browser in ways we couldn’t with Chrome. You can also view the video directly inside Firefox, by going to Tools > Add-ons > Get Add-ons > Click “Learn More” / “Learn More About Add-ons”.

Popcorn.js

For anyone unfamiliar, Mozilla’s Popcorn.js makes it easy to interact with video online using JavaScript. For anyone who has worked with jQuery or a similar library, it’s a breeze to pick up. Popcorn.js is the code part of Mozilla’s Popcorn, which is a larger project that aims to supplement video and audio with rich web content.

Setting up Popcorn.js is incredibly simple:

var pop = Popcorn('#promo-video');

Popcorn.js has a plugin called code, which lets you run arbitrary code when the video hits certain points.

pop.code({'start': start, 'end': end, 'onStart': function(){…}, 'onEnd': function(){…});

That’s basically all you need to know to get started with Popcorn.js. Everything else we did used regular JavaScript.

Like I mentioned, we only used one of the many plugins. Popcorn.js is incredibly powerful, and our video barely scratched the surface of what Popcorn.js plugins can do.

Showing Add-ons

We started by making add-ons fade in on the right side of the video when they are mentioned.

Note: You will only see this in the video if your browser window is wide enough.

Previewing Personas

We’re off to a good start. So far, we have add-ons showing up when they’re mentioned in the video. Now, wouldn’t it be cool if the video could interact with Firefox?

At one point the video talks about Personas, which are basically skins or backgrounds for Firefox. Since it would be incredibly annoying if any site you visited had the ability to skin your Firefox, the ability to trigger a preview is whitelisted to just the Mozilla add-ons sites.

Note: You can add other sites to this whitelist by going into Preferences > Security > Exceptions.

We thought it would be fun to style the browser with the Persona being mentioned — especially since we’re the only site that is allowed by the browser to do this. We simply fire an event when the video hits a certain point. To make sure the Personas are displayed instantly, we pre-load the Personas when you first click “Play”.

Blown Away

Later in the video, we see a plane fly across the screen. The papers and other items on the desk start to blow away as this happens. If you scroll down a bit, the add-ons below the video also start to flutter and are eventually caught in a JavaScript-y wind and blown away.

This is done using -moz-animation and -moz-transform: matrix(). Each add-on is given a class of “flutter” (starting at a staggered interval), and the add-ons flutter back and forth until they eventually “blow away”.

This isn’t Firefox-only; I just left out the Webkit code in the example for the sake of brevity.

In the next scene, both the backdrop in the video and the add-ons on the site drop back into place.

Other Things to Watch For

During the scene with the cameras, the page flashes as though a picture with a flash was being taken. The elements are simply hidden, then faded back in.

When the video talks about how add-ons can translate a page, some of the text on the page is translated along with the text in the video. (Note: This only works if you have fewer than three add-ons installed or you watch using this link. Otherwise, the top bar is not displayed.)

Uh-oh

We learned a few things along the way. We used popcorn-complete.js, which we didn’t realize loaded a few CSS, JS and SWF files off of popcornjs.org’s servers when the file was loaded. We mistakenly DOS’d popcornjs.org, which in turn made the add-ons site load slowly. Thanks to Jon Buckley and Rick Waldron, we were able to figure out what was going on and push a fix fairly quickly.

I also owe Chris and Krupa a big hug when I get back to the Mozilla office — they helped me out a ton by finding and fixing a bunch of bugs as we were pushing to production, since I was an unfortunate 30,000 feet in the air as the video was going live.

Looking Ahead

I have no doubt that Popcorn.js is going to become huge as HTML5 video takes off. News sites will use it to show relevant links and tweets, fashion sites will link to online stores as a celebrity is shown wearing a dress or shirt, video sites will provide closed captioning and annotations, media sites will use it to sync viewing across multiple computers — the possibilities are endless. I cannot wait to see what Popcorn powered videos people make next.

 Watch The Video
On the page, click “Learn More” to play the video.

Since the add-ons site is open source, be sure to view the code if you’re interested.

As for my family: not all is lost. The next morning I got an excited text from my cousin, who had watched the video. “Guess what I just did! I installed one of those picture addons on the top!” Sure, it pains me a bit that a Justin Bieber Persona now graces her Firefox — but hey, it’s a step in the right direction.

 


An optimization story with Django – one thousand times faster!

I’ve been working on a Django project that is going to be squeezed in as an external application under Pootle which is our tech of choice for the Mozilla Verbatim site which is where all awesome translations happen for Mozilla web projects. (translations for Firefox, Thunderbird, etc. is done by another project which I’ll blog about more in the next couple of weeks).

In Pootle, there are “Profiles” and that’s where the name and quick stats on contributions is stored. However, what I needed to do was to actually go through every “Suggestion”, “Submission” and “Review” and figure out who has contributed what on what project and language. Every such model has an overridden object manager that always joins on the “Users” model thus causing a ‘INNER JOIN’ on every simple data query. Thus this would build up an accurate page of who has contributed to what on the fly.

There are 14 projects, 99 languages, 847 users, 37,681 suggestions (and reviews) and lastly 127,226 submissions. The task is to group these first by language and then by project and lastly for each project, list the contributors names in alphabetical order. After some getting used to the data model and where things are located this wasn’t difficult. I built a prototype that visually represents this and showed it to the stakeholder who wanted this built.
The only problem is that due to the way the Django ORM and Pootle does this is that this takes on average: 11 seconds and involves 2,544 SQL queries. Yikes!

After some optimization (helped by my colleague Rob Hudson’s wonderful django-debug-toolbar) I was able to turn 11 seconds into 10 milliseconds (ie. one thousand times faster) and reduce the number of queries from 2,544 to 7! All it takes is just having to lessen some of the convenience of Django’s otherwise super practical ORM and get closer to the “SQL of things”. The code doesn’t express anything in raw SQL and everything is in Python with some very specific loops.

Why 7 queries and not 1, you might wonder. The reason is that the data I need to get to is spread across 7 different tables. Because some of them are large and joins are difficult what I do instead is pick only exactly what I want and need and nothing more. The logic kinda goes like this:

    • Fetch all IDs and their Name (e.g. 52 and “Arabic”)
    • For relevant tables with foreign keys, only fetch its ID, the foreign key ID and keep this in a list or dictionary whose structure implies information
    • Sort and re-organize these lists and dictionaries in Python
    • Summarize and replace the IDs with the Names for the final template rendering

I think the lesson here is that if your structures are big but not so big that you need a map/reduce job then getting closer to the raw SQL queries and doing the structure juggling in memory, you don’t need to replace your relational databases with NoSQL or typed raw SQL. Django’s ORM is fantastic. It’s slow because it chooses to be convenient but if it needs to be fast it’s just a few slight API calls away. If you’re curious, check out the code on Github.