Migrating Air Mozilla from MySQL to PostgreSQL

Peter Bengtsson

Before we dig into the how let’s take a look at the why.

From the beginning, Air Mozilla has been a straight forward Django project that uses the ORM without requiring any database specific features. It didn’t really matter what database you used. Here at Mozilla, we currently prefer MySQL because we have a rock solid and mature infrastructure set up around running it. (Thank you database team!)

This week we launched full-text search in Air Mozilla. Here’s an example search. PostgreSQL supports very powerful features specifically for full-text search, including stemming, highlighting, ranking and custom dictionaries. (Note: MySQL has full-text search indexing too as of MySQL 5.6 but it does not yet support stemming or highlighting).

So how did we migrate the database? In short, this tool: py-mysql2pgsql. What the tool does internally is that it connects to both MySQL and PostgreSQL and reads one table and record at a time to convert over to PostgreSQL. You can check out the code on github.com/philipsoutham/py-mysql2pgsql/.

To run it, all you have to do is fill in the connection details into a YAML file for both MySQL and PostgreSQL and you should now have a working “clone”.

There was one caveat though that irked me. MySQL does not support timestamps with time zone and PostgreSQL does. Django can work around this by applying the time zone with a Django settings variable. By having the time zone information in the database, we don’t have to fake the time zone information any more. It’s also bound to be more performant because you’ve moved the conversion to aware date times nearer to the database. To make this change, we wrote a simple conversion script that you can afterwards throw into PostgreSQL. You can see the rest of the instructions for the migration here.

And here’s a little bonus; the time it took to actually run the migration was approximately 10-15 seconds to migrate over 25,000 rows across 42 tables. That’s connecting to a MySQL and a PostgreSQL in two separate different locations in the same data center.

In a follow-up post I will try to explain more about how we do the full-text search in PostgreSQL with Django.

Beer and Tell – September 2013 Edition

Michael Kelly

Gather ’round, children. Your distant cousin mkelly is going to share a tale of excitement and mystery, of heroes and villains, of action and adventure.

That’s right, it’s the Beer and Tell Recap! You can also check out the wiki page or the recording.


 Appmaker

Screen Shot 2013-10-03 at 10.57.19Simon Wex and Robert Richter from the Mozilla Foundation presented Appmaker, an experiment into whether we can make it fun for non-developers to quickly make working apps. Widgets send “mail” to other widgets, which triggers them to do stuff like display cat pictures or take a photo. You can track development of Appmaker on their Github repo.

diecast

My own Beer and Tell project is diecast, a grunt-init template for single-page frontend apps. diecast sets you up with Grunt commands for building and publishing your site to Github Pages, and includes require.js for JavaScript module loading and LESS for CSS preprocessing. It also uses Bower for downloading JavaScript libraries that you want to use. It’s a cornucopia of JavaScript buzzwords!

nodeschool.io

Screen Shot 2013-10-03 at 10.56.48Brian Brennan showed us nodeschool.io, a community of learning focused on node.js. It’s focused around terminal-based challenges where you write code to solve realistic problems. Planned improvements include user accounts and open badge support.

OpenCL in the Browser

Scott Michaud presented a super-secret project (no links, sorry) that demoed a software renderer powered by OpenCL code running in Firefox. Presumably he was using the WebCL prototype plugin released by Nokia Research recently. Check it out!

Dennis Dubstep

While he was absent and thus didn’t present, Will Khan-Greene still added a screenshot of SUMO localized to dubstep on the wiki. This was achieved by using a dubstep locale in dennis, a set of localization tools that can, among other things, help you test out how your site looks with excessively large strings that may come from certain languages.

Crass

Matt Basta told us about Crass, a CSS minifier that parses CSS instead of applying transformations, as most minifiers do. This allows it to perform optimizations that most other minifiers can’t, such as reordering properties and transforming values. Crass can also pretty-print the parsed CSS and is written in both Python and JavaScript.

Panopticon

Basta also shared Panopticon (app is down as of writing), which is sort of like Skype and IRC combined. You join a room, select a user in that room, and get to see an animated GIF captured from their webcam, live. There’s a Github repository as well, for those who are interested or confused.

Updated Nunjucks Documentation

James Long demoed an upcoming update to the documentation for Nunjucks, a jinja2-inspired templating system for JavaScript. Keep an eye out for the update, which includes a full overview of the template language and API!

BATCH

Screen Shot 2013-10-03 at 19.28.10Finally, Michael Cooper presented a short game called BATCH, a programming game written in JavaScript. The main purpose of the game was to see if he could write a Beer and Tell project starting at Noon on the day of the event. I highly recommend that other people try this as well, if for nothing else than to stuff the Beer and Tell project list for next month.


See ya’ll next month!

Where Mozilla.org and Firefox Intersect

Holly Habstritt Gaal

written with Chris More and Jennifer Bertsch

Engagement’s Web Productions team has grown from a small team of technical project managers, to a multidisciplinary web development team that initiates projects, uses metrics and qualitative testing to learn from our users, and has an iterative approach to web development. We’ve had a chance to see the influence that this growth, paired with the collaboration of teams across Mozilla, has had on our work. We can be pre-emptive instead of reactionary, share our knowledge and tools, and facilitate design process and collaboration. We would like to reintroduce ourselves to Mozilla and the UX community, expose where our team intersects with the user experience of our products, and invite you to collaborate with us.

Not your typical web product
Mozilla.org does not solely exist for marketing our products. It is unique from most web sites in that to support our products and users when they need us, we must stay inline with the roadmaps and release cycles of our products. For example, there are many touch points with our users that take place on Mozilla.org, some of which are part of the onboarding process. Onboarding is more than just downloading our products as it extends to the first “unboxing” experience, updating Firefox, and sharing helpful information about new product features. This ultimately contributes to retention and an understanding of our Firefox and Mozilla brands.

In our user tests, we’ve found that users are more likely to respond positively when they have an expectation of both when and how a message is delivered. For Mozilla.org and our products, this expectation can be set by previewing an upcoming feature or new design on Mozilla.org for Firefox users. It can also be handled with a consistent pattern for how we present content updates, notifications, and new features across all of our products. The WebProd team doesn’t accomplish this alone and this is one example of how our users can benefit by our teams staying connected.

Staying connected: the intersection of our roles and roadmaps
What I’ve found at Mozilla is that separate teams are often working on similar challenges and share common goals. Collaborating across teams has been a great way to meet and learn from each other and is key to addressing our intersecting issues efficiently while ultimately creating a better end product. Most recently we have worked with SUMO to stay better aligned on presenting Firefox help messaging and we have also been collaborating on a cross-team effort to improve the First Run and Update experiences.

At Mozilla we are all part of the chain of reactions that results in what our users experience. The WebProd team has been keeping the following in mind to better support Firefox users:

  • Align our websites to product roadmaps so we can offer support to our end users
  • Optimize user onboarding flows
  • Work in parallel with Engagement and Product teams’ goals
  • Ensure website content is localized in many languages
  • Complete migration of all legacy pages to Bedrock and our Sandstone theme, which is responsive by nature.
  • Support users on any device or operating system
  • Evaluate > Test > Improve

A significant way we can all support our users is to recognize the intersections between our teams at Mozilla and the overlapping initiatives in our roadmaps. If the WebProd team can collaborate with you to create a better experience for our users, don’t hesitate to reach out to us.

We’re easy to find!

DXR gets faster hardware, VCS integration, and snazzier indexing

Erik Rose

12

DXR is Mozilla’s fast, full-featured code search tool for doing structured queries, free-text searches, and regex matching on huge codebases, like Firefox’s. Since my last post, we’ve completely replaced the production hardware, integrated with VCSs, and made scads of indexing improvements. The last quarter’s highlights include…

  • New stage and production hardware, with our own dedicated build box so we can support multiple codebases
  • Nightly updates from the mozilla-central tree, for both stage and prod
  • No more hours of downtime when a mozilla-central build fails
  • Blame, log, diff, and raw-file links in the sidebar
  • Much better JS syntax highlighting
  • Linkifying much more C++ template stuff: references to class templates, template params, and base classes of class templates when the base class is also a template and dependent on the type params of the derived class template (phew!)
  • Searching for namespaces and namespace aliases
  • Better finding of C definitions and declarations
  • Clang 3.3 support

Tell us what you want.

Now we want to hear from you. We’ve got a big, juicy fourth quarter coming up, and we want to make you happy. What would you, as a current or potential DXR user, like to see happen? Some ideas already high on our list are…

  • Indexing multiple trees, like Aurora and comm-central, eventually targeting the full list from MXR
  • A UI refit that will resolve client-side bugs, improve consistency, and add power. Take a look at the wireframes!
  • Structured search for JS: finding function definitions and calls, variable refs, and so on

What are your highest priorities? What would help you hack better on Mozilla code today? Leave a comment about what’s most important to you, whether it’s in the above list or not, and we will build our Q4 goals based on what you say.

Also, I’ll be manning a table at the Innovation Fair at the Mozilla Summit in Santa Clara. Stop by and make DXR wishes in person!

Finally, thanks to the people who make DXR possible: fubar, for all his ops work; and jcranmer, abbeyj, nrc, Bruce Stephens, jonasac, and nicolaisi, who keep the patches rolling in faster than I can review them!

2013 Web Dev Intern Recap

Lonnen

With the 2013 intern season winding down, we’d like to highlight some of the intern contributions to Mozilla Web Dev projects. This year five interns from four countries spent a few months at our Mountain View office building amazing things:

Ashish Dubey created a communication dashboard in the Firefox Markeplace for archiving communications between app developers and reviewers. The dashboard uses email notifications and syncs email responses, providing some flexibility for users. Ashish also created MarketPlay, an interactive sandbox for exploring the Firefox Marketplace API.

Derek Ries created Kanbanzilla, a stand-alone app that uses the Bugzilla API to provide a Kanban interface to Mozilla’s bug tracker. This allows Mozilla teams to use Kanban without having to keep an external system in sync with Bugzilla.

Shu Hao Wu created a webapp for Mozilla’s support website (SUMO). The app provides offline access to the SUMO troubleshooting wiki for FFOS with fast, full text search that works well across SUMO’s localizations. Shu Hao also contributed to Mozilla’s crash reporter, where he added automated identification of rapidly growing (“explosive”) crash signatures.

Leon Zhang built the first iteration of the Brick web components library. Brick is a javascript library that provides common UI components for web development. Brick builds on the x-tags polyfill to offer the benefits of the emerging Web Components standard today across browser platforms.

Tony Young was a spring intern who refactored several modules of the distributed system powering Mozilla’s crash reporter. He wrote a faster filesystem module, brought consistency to the hbase module, and made storage modules composable to make Socorro easier to install and enable quicker development of new features and components in the future.

Many thanks and well wishes to our interns as they return to school! Also, a great big thank you to Mozilla University Recruiting, for furnishing us with such an excellent group this year!

All of this year’s intern presentations are available on Air Mozilla. If you’re interested in a Mozilla internship, you can get more information from university recruiting, see open positions on our careers page or meet with some Mozillians at a campus near you.

Firefox Marketplace: June 28th – July 18th

Andy McKay

This is a regular post focusing on the status of the Firefox Marketplace.

  • Total bugs open: 559
  • Total bugs opened: 315
  • Total bugs closed: 235

Some specific changes of note:

  • Separated out the API between installing a paid app and a free app (bug 886568).
  • Packaged app receipts now include the origin from the manifest (bug 867265).
  • User interface created for the communication dashboard (bug 885391).
  • Fireplace API moved off to its own URL namespace (bug 875536).
  • App features added to the API (bug 881765).
  • Developer agreements upgraded (bug 891124).
  • Installing of free or paid apps with $0 cost all sorted out (bug 889501).
  • Keyboard auto-suggest disabled for search (bug 890569).
  • Newsletter sign up added to the Marketplace (bug 874932).
  • Payment documentation added to the marketplace (bug 839292).
  • Payment PIN screens localized (bug 885551).
  • Don’t validate apps where the app manifest does an off-origin redirect (bug 849548).
  • Track how many searches return no results (bug 890314).
  • Developers notified of unsupported locales (bug 836147).
  • Fix queue for “public waiting” apps (bug 868526).
  • App name translations now picked up during initial submit (bug 893855).
  • Fixed app installation on Linux (a bug 876293).
  • Polish language search analyzer added (bug 887944).
  • Packaged app origins must be unique (bug 878105).
  • Fix the PIN interface to work well in multiple locales (bug 887875).
  • Allow free with in-app payments apps (bug 857733).
  • Allow localized in-app payments (bug 892612).

Most more important of all, we shipped the Marketplace on phones.

Payments were enabled in Spain and Poland for the releases and apps have been successfully bought.

This was a huge milestone for the Marketplace team who did a great deal of work to get to this point. Thank you to everyone who contributed.

Web Productions Project Updates — July 2013

Chris More

The following is the July edition of the Web Productions accomplishments and upcoming projects.

Recent Key Accomplishments

  • Launched Firefox OS consumer website on-time (UX improvements coming soon!)
  • Migrated lots of legacy pages from the old mozilla.org website to new homes! (bye bye old designs, hello sandstone theme!)
  • Launched improved Mozillians homepage
  • Added voting to Mozilla Reps
  • Released improved newsletter subscription center (More improvements in the works!)
  • Released advanced Google Analytics metrics on Firefox Marketplace (tied to business objectives and KPIs)
  • Developed improved about:home snippet service (Waiting on IT for infrastructure improvements to release)
  • Researched and designed UX improvements to MDN
  • Refined development process and tools

Snapshot of Upcoming Projects

More Info

Have any questions? Chat with us in IRC in #www or #webprod.

Kanban & Web Productions: Lessons Learned

Benjamin Sternthal

1

Kanban has been making in-roads at Mozilla, and this is no small feat considering the size and complexity of the organization. I’d like to think our team has influenced folks, either to adopt Kanban, or to at least think about alternative development methodologies. Below is the history of how Web Productions came to use Kanban and the lessons we learned along the way.

Continue reading …

Firefox Marketplace: June 14th – June 27th

Andy McKay

1

This is a regular post focusing on the status of the Firefox Marketplace.

  • Total bugs open: 577
  • Total bugs opened: 244
  • Total bugs closed: 193

Some specific changes of note:

  • Some APIs have been copied to a new namespace for Fireplace because they are unique to Fireplace (bug 875547, bug 875548)
  • Receipt reissue endpoint reserved for when receipt replacement is possible (bug 881788)
  • Reviewer to developer communication dashboard created (bug 879469)
  • Multiple sorting for search results added to the API (bug 867848)
  • Lots and lots of crufty code deleted, which is a big deal to us (bug 881910 and others)
  • Fixes when accessing a pending app (bug 870020)
  • Multiple carrier billing payment fixes in some of the target countries
  • Fixes for Persona changes (bug 884360 and bug 884122)
  • Editing and cancelling reviews work (bug 885297)
  • API now shows created date and number of downloads (bug 867839)
  • Screenshot navigation much smoother (bug 887316)
  • Developed parameter on search API now consistent (bug 886385)
  • Payment setup in the Developer Hub now shows regions for the payments (bug 884443)
  • Cope nicely in the Marketplace with in-app payments that can’t be processed (bug 878216)

The performance on the search API was drastically improved with a few changes. Search performance improvement (lower is better):

graphite-search

Unfortunately, I wasn’t able to attend the “show and tell” meetings recently so there aren’t any updates from there.

Lightning Talk Bonus Round

Michael Kelly

It was raining when the idea first came to me. Deep within the disappointingly well-lit halls of my local coworking spot, I sat in a meditative state as I contemplated the problem of sharing knowledge among the Web Productions team. As my mind drifted in and out of various realities, I came upon a world in which the only television show was the Price is Right. Taking inspiration from the wondrous paradise that this world was, I decided the proper solution was to create a monthly event: Lightning Talk Bonus Round!

Essentially, once a month the Web Productions team gets together to share short 5-15 minute talks on any subject we desire, from programming languages to design principles to mycology. We’re still working out the format and technical details, so there are no recordings available… yet. In the future, we hope to record the event and open it up to the community.

The first Lightning Talk Bonus Round was on June 3rd, 2013. There is a wiki page with links to the notes and details about each of the talks.

Craig Cook – Design Principles

Craig Cook gave a shortened version of a longer talk focused on the basics of Design Principles. He discussed controlling the balance between elements on a page to indicate which elements are more important than others. He also discussed using proximity, repetition, and contrast to help visually distinguish elements and establish the hierarchy of important on a page through visual cues. You can check out his slides on GitHub.

Screen shot 2013-06-18 at 2.53.53 PM

Giorgos Logiotatidis – Django Anonymizer

Next, Giorgos showed us Django Anonymizer, which is a Django app to help you anonymize existing data in the database. As opposed to other methods, such as maintaining fixtures or generating random data, with Django Anonymizer you’d start with a copy of your production data and configure the app to anonymize it, leaving you with data that is anonymous but still looks similar to the data you’d see in production.

This is especially useful in situations where production data may end up with interesting relationships between data that are hard to replicate when generating random data from the start. The Django Anonymizer project is available on PyPI. You can also see Giorgos’s notes in a gist.

Paul McLanahan – virtualenvwrapper and hub

Lastly, Paul talked about two tools that he finds useful during day-to-day development: virtualenvwrapper and hub. virtualenvwrapper is a set of extensions to virtualenv, which creates isolated Python environments for projects that need specific versions of Python or other libraries. The extensions come in the form of commands that help you quickly create, activate, and delete virtual environments. There are also hooks that let you execute certain commands when environments are created, activated, or deleted.

hub is a command-line wrapper for git that has extra commands tailored for working with GitHub. You can clone a repository using just a username and repository name, automatically create a fork on your GitHub account, or even create a pull request for a branch, all directly from the command prompt.

Screen shot 2013-06-18 at 4.45.52 PM

Great Success!

As I gaze over the majestic glory that is the aftermath of Lightning Talk Bonus Round, I realize that we have only just started. My work shall continue, on the first Monday of each month at 10 AM Pacific, in the hopes that someday we will be able to avoid all this knowledge sharing and pool together in a mindspace where thoughts mingle like so many fish in an ocean of web development.

Until then, we shall have to do what we can in meatspace. See you next month!