mrz's noise

noise from a mozilla IT/Operations wrangler

I dare you.

For the past several years in December I stop shaving. I join an ever growing community, a cadre of co-workers, of friends, of strangers.  A Tribe.  For 31 days we rejoice in not shaving, in letting our scraggly faces grow.

And then it’s over.

I dare you to make me keep going.

Those who know me know at least two things.

First.  Year Up.

Ever since being introduced to Year Up in 2009 I’ve been a passionate supporter.  Year Up is working hard to shrink the opportunity divide, to provide young adults with access to education and tools that today’s world needs for a successful career.

I wrote about Year Up but it’s worth repeating – the young folks I’ve encountered from Year Up embody intensity and determination for being more than they are today.  They are not satisfied with the status quo.  They are tomorrow’s leaders.  They are my inspiration.  They make me cry.  They make me laugh.  They will realize their full potential in ways I can not yet imagine.

Second.  Mozilla.

Mozilla is many things to many people.  For me, it’s a Mission and a Community.  Part of that Mission is Mozilla WebMaker.  Mozilla WebMaker is working hard to move people from simply using the web to making the web, “to increase their understanding of the web, take greater control of their online lives and create a more web literate planet.”  Without exaggeration, Mozilla is creating the next generation of web makers.

I Dare You

Imagine what happens if you teach tomorrow’s leaders how to create a more web literate planet?  Imagine if we can leverage the work Year Up is doing to bridge the skills & opportunity gap with tomorrow’s skills and WebMaking?

Imagine if everyone could realize their full potential.

Imagine if you played a small part in making that happen?

Can I combine those two passions – Year Up & Mozilla – and create something bigger than myself?  Can I put myself out there in a very public way?  In a way where you can laugh at me and help make a difference at the same time?

Can you keep me in this beard until March or July?  I dare you.

Dare accepted?

Help me raise money for Year Up by going to their donation site and write in “mrz beard” in the “Please allocate my donation to” blank.

Every dollar contributed will make an immediate impact on the lives of young adults, helping them reach their potential. Every dollar contributed will help teach WebMaking skills to tomorrow’s leaders.  Every $500 you help me raise is one more day I don’t shave.

Keep me in this beard until March, or June or even next December – I dare you.

Air Mozilla Reboot, Phase I

With very little fanfare we rolled out a complete refresh of the Air Mozilla website.

As with everything, this was far from a solo act and I have many many people to thank in helping make this happen.  Thank you.

Air Mozilla

Air Mozilla is where we tell the Mozilla story in video, where, we hope, the open web meets open video.

It’s seen itself evolve several times but it’s core mission has been consistent – share.  Share what we do with the whole of the Mozilla Community.

What started out as just a way to share the Mozilla Weekly Project Meeting with the community, has grown into the foundation for a platform to share nearly everything we do.

The website has never had a lot of attention focused on it.  In October, we set out to change that.  What you see today is the result of several months of work and effort and a now iconic graphic element that first made its appearance at MozCamp Europe.

Air Mozilla is more than just the website.  It is,

  • Two full time paid staff who manage the AV and streaming from Moz Spaces and spend time helping the Mozilla Community stream from remote locations.
  • Two developers working hard to push WebM forward.

What’s next?

We work on Phase 2.  Here’s a glimpse at our goals for this quarter:

  • Prove that WebM can work at scale
    • Simulcast Mozilla events in WebM & Flash
    • Start Air Mozilla Artist Series
  • (phase 2) (bug 712717)
  • Stand up Digital Asset Manager
  • Issue Air Mozilla Open Badges
  • Live streaming from London & Vancouver
  • Big Blue Button integration in San Francisco & Toronto

Want to get involved?

If you’re a skilled web developer, interested in open video or interested film/multimedia, you should be a part of Air Mozilla.

The new Air Mozilla.

Zimbra & Mozilla email, 4 months later

Four months ago was a very tough time in Operations.  We suffered a catastrophic disk array failure on Mozilla’s mail server (I blogged about it too).  A series of mistakes kept email offline for two days.  This was the worst I’ve ever felt, both professionally and personally.

Fast forward to today.  So. Much. Better.

We learned.  We researched.  We re-organized ourselves.  Much like The Six Million Dollar Man, we rebuilt it better.

justdave posted his account, “Re-imagining Zimbra email at Mozilla” but I wanted to add my own color.


During my interview at Mozilla in 2006, I was asked a bunch of questions about Zimbra.  First I had heard of it.  By the time I started I had learned quite a bit about Zimbra. Back in 2006, email was hosted externally and we began the process of moving email back in house.  The company hosting email couldn’t provide SSL and wasn’t doing all the groupware things we needed.

Post Zimbra-gate (December)

I was mentally done with email. I looked at simply outsourcing.  I looked at hosted Zimbra, hosted Exchange, hosted whatever.

We Mozillians, we’re a unique group.

  • We want to use the IMAP client of our choice. Some of us just want to use the web interface. Others prefer Microsoft Outlook. Or Thunderbird, or or Postbox or mutt or pine or Sparrow or …
  • Calendaring is just as complex.
  • We need to support a wide number of mobile devices – iOS, Android, Blackberry, devices that support Microsoft’s ActiveSync – with both email and calendaring.
  • Some use Zimbra’s document sharing/storage
  • We need something that supports IMAP, ActiveSync, CalDAV, CardDAV.

We looked at what others at our scale and beyond our scale use for email.  Oracle uses Zimbra.  Comcast uses Zimbra. At. Scale.

We talked to others hosting their corporate email with Google Apps (and their 15-person staff managing their Google Apps mail!). We learned that deploying Exchange requires a move from OpenLDAP to Active Directory and a particular skill set that we don’t have in house.

Moving Forward

This incident highlighted the need to have a team focused on infrastructure.  Our primary focus (and priorities) always tend to lean towards various Mozilla web properties or developer services.

So we did two things –

  1. Broke up a fairly flat Operations group and created an Infrastructure Operations team (and a couple others) to focus on services like email & LDAP, to name a few.
  2. Built a new environment for services that, when break, cause work stoppage, cause a line to form behind my desk.  This Hyper Critical Infrastructure, or HCI, is isolated from the rest of the production environment, has different change control processes and is meant to hit as many “9s” as we can hit.  It’s a very different way of planning than we had done in the past. 

    This technology stack uses more corporate/enterprise technology than we’re used to using at Mozilla.

HCI Today

HCI straddles two high density, (~15kW) racks.  It’s only relation to the rest of Mozilla production network is two 10GbE fiber drops from the network core.

HCI has it’s own Juniper SRX 1440 firewalls.  Its own Juniper EX4500 switching.  Its own NetApp FAS3270.  Its own 5 node VMware ESX cluster, each machine having 2x 6-core Xeons & 192GB RAM.

In a couple months, services here will be replicated to SCL3 using various NetApp & VMware technologies.

We had planned to have HCI in production by the end of February but no one wanted to rush this (plus someone decided to have a baby).

Instead we slipped that to the last week of March and I’m glad we did.  We consulted with Zimbra and others.  We sent Desktop & InfraOps to training. We tuned and fine tuned.

Zimbra Today

We have mailboxes spread across seven mailbox servers and understand the metrics we’ll use to determine when to add more mailbox servers.

We migrated 1002 mailboxes from San Jose to Phoenix without anyone noticing, without any user impact, in just a couple days.  In fact, we didn’t mention it until we were done.

We have instrumentation and trending and alerting on everything we could think of.

What’s next?

All is for naught without learning. We learned a lot and we’ve changed how we operate as a team.

Once bitten, twice shy.

Mozilla IT & Growing Community

Last week Lukas blogged about her thoughts on David Eave’s community lifecyle audit. I jokingly told her that I felt she called me out.

Until we require Directors to create annual and quarterly goals that include measurable goals around volunteer growth, retention, participation, and effectiveness we will only see people (like myself) trying to do this “off the corner of their desks” which means it’s not a part of your paid work and thus less likely to be sustainable and effective.

Community today

Two points:
  1. I’ve made it clear to my teams that Community is a focus in 2012. I want each of them to spend some percentage of their work time engaging with the Community.
  2. I, as a Mozillian who carries the title “Director IT/Infrastructure & Operations”, have created annual & quarterly goals that involve the Community.

I’ll restate the commitment I made back in October:

My own personal goals by the end of 2012 are:

  • to have 5-10 volunteer Community Sysadmins actively helping run Mozilla’s network and servers.
  • to have a vibrant Community IT group…
  • to have a premiere source for open source video technology, a site where the Mozilla Community can find, share and create video content

Community, tomorrow?

Back to Lukas.

She’s right and while it may not seem obvious, Mozilla IT started including Community specific goals since last quarter.

2011 Q4 Goals – Grow Mozilla
  1. IT Pivot to Open. Start discussions to generate 2012 plans.
    1. [DONE] Community IT
    2. [DONE] Community Volunteer Sysadmins
    3. [DONE] Air Mozilla
2012 Q1 Goals – Grow Mozilla
  1. Community IT. Empower the wider Mozilla Community.
    1. Spin up infrastructure to support Open Badges
    2. Enable community to join Yammer
    3. Develop infrastructure for Community website hosting (Joint with ReMo)
  2. Air Mozilla
    1. Website Reboot (bug 712717)
    2. Mobile Air Mozilla Kit. Develop a reference kit to enable anyone to stream a live event through Air Mozilla.
    3. Host one Community Air Mozilla training event
    4. Incorporate one Open Badge concept around video tagging or event streaming/recording
    5. Vidyo & multi-tenancy with


I keep coming back to this powerful but simple idea:

Be fierce. No one will build into the Internet the kinds of things we want to build.

This can not be done alone.

Step 3.05: Overwhelmed

On November 28 I posted about our efforts to get more volunteers into Mozilla IT. A lot has happened in the background since and I want to let you know what’s happening (and why it’s taking so long!).

O. M. G.

The response and interest has been overwhelming.

I listed five different positions volunteers could “apply” to. I chose to use our exising applicant tracking system, Jobvite (I already have enough wheels, don’t need to invent another) and many of you have already expressed interested and applied.

Here’s a run down of responses so far:

Volunteer Role Responses
Mozilla Mirror Administrator 30
Desktop Padawan 5
Air Mozilla Stage Hand 4
Mozilla Foundation SysAdmin 14
Data Center Operations Engineer 4

What’s next?

    • There’s a lot of people to talk to and we’re slowing scheduling time to chat.
    • I talked about the Mozilla IT Volunteer Agreement. We’re continuing to refine it. With Gerv’s and Jishnu’s help we’ve carried this conversation to mozilla.governance.
    • We are double-checking to make sure we understand issues around volunteers and various government regulations.

Unfortunately this means we’re moving a but slower than I had hoped. If you’ve already expressed interested, that’s fantastic and we’ll be talking to you soon.

Bear with – I think we’re in slightly uncharted territory!

Step 3: How can you volunteer @ Mozilla IT?

In most ways, right now at least, you can’t. We’re too closed. It’s like I said in my first blog post,

IT is generally closed. Mozilla is not. There’s a incredible disconnect there. How do we leverage the expertise of the Community in running some of the busiest websites in the world?

In my travels over the past year I’ve met a number of passionate volunteers with IT skills who are looking for different ways to volunteer and contribute to Mozilla.  In the past two months, that list has exploded.

I’ve talked about ways Mozilla IT is trying to help the community, how we are trying to be your IT (we’re still talking about it too!).

I’ve talked about how we want to reboot Air Mozilla, how we want to open video and make it possible for more people to tell the Mozilla story in video.

But Mozilla IT is still closed.

Help me change it?

I want to illustrate what we want to do.

The best way to do that is to share a little open video and popcorn.js, with some help from Popcorn Maker. (This is what happens when you go to a Mozilla Festival. See how great the open web is?)

Watch this to see how we want to pivot to open.


Good question.  I think I need your help to figure this out. It’s going to feel weird and uncomfortable for us.  Of all the steps we set out to do nearly two months ago, this has been the most challenging.  There are so many processes to work out.

  • What sort of agreement should volunteer Mozilla IT sign? A code of conduct? There are some parts of the infrastructure that must remain secure and secret even as we strive to be open.
  • How do we build the trust necessary to give someone root access?
  • How do we on-board new Mozilla IT volunteers?  Does everyone get root access on day one?  Is there some graduated process?  What is it?
  • Do we host onsite (or remote) training events to teach you about our tools and processes?

Today, as a code contributor, we ask you to sign a Committer’s Agreement.  It’s a simple document that shows you understand what it means to contribute code to Mozilla and understand our legal requirements.

As part of Mozilla IT,  you’ll have access to some pretty mission critical systems. I invite you to take a look at the Mozilla IT Agreement and share your feedback with us.  It’s meant to be a lightweight agreement similar to the Committer’s Agreement.

Want to get involved?

We’re doing a lot of this thinking out in the open at and I invite you to join and participate –

Preparing for 2012

I want to reiterate two of my goals I mentioned in my first post nearly two months ago.

My own personal goals by the end of 2012 are:

  • to have 5-10 volunteer Community Sysadmins actively helping run Mozilla’s network and servers.
  • to have a vibrant Community IT group…

I made the comment that it felt like the most ambitious thing we’ve done.  It probably still is but in two months we’ve shifted our way of thinking, took Air Mozilla Mobile on the road and have a long list of things you need from us.

2012 will be fun.

Mozilla’s new Santa Clara data center

Short story:

We’re building out a 1MW data center presence in Santa Clara, CA.

Our hosting needs are radically different than they were five years ago when I started and we had a couple racks in Sunnyvale.

We now have six data centers around the world, three of which are within fifteen miles of Mountain View and consume about 455kW of power.

We’ve reached a point where it makes more sense to look at wholesale data center space (the same sort of space that the Facebooks, the Googles & the Zyngas use).

We spent a good part of the first half of 2011 researching various locations in the area before deciding to partner with Vantage Data Centers.

Construction started about a month ago and we’re on track to start turning on services in January!

Being open

Taking inspiration from our own creative design process and even our development process, we’re going to build out this data center in a uniquely Mozilla way.

We’ll be as open as we can about the whole process. Building a data center like this is something new for Mozilla and this will give the community an opportunity to see how we do it.

Here’s why this matters. Facebook/Google – those guys’ scale only applies to 1% of the world. The number of content providers that can shut down a whole data center because it got too hot outside and not even flinch can be counted on one hand.

Our scale is much more inline with 90% of the world. We have a lot of expertise in doing scale. We have a lot of expertise in running data centers that apply to most of the world. This is where we get to share how we do it & why we do it the way we do it.

Long story:

This isn’t an mrz effort so I’m not going to talk about it here.

Instead, we’ll be talking more about this at the Project: SCL3 blog. In fact, you should go there right now!

Step 2.1: #MozFest, Air Mozilla & London

What. A. Weekend.

My first real stay in London was punctuated by Mozilla Festival. And what a weekend it was. Radically different DNA than any other Mozilla event I’ve been to.

In many ways it reminded me of the Hacks/Hackers meetup I was at last spring in Buenos Aires. The combination of the open web and journalism is pretty energetic, a mixing of common values and beliefs.

It’s like the MoJo wiki says,

“Journalism and the open web are built on common values. And, like the web, the future of journalism lies in universal access and participation. ”

I meet interesting people (@GemmaCocker, @RadioKate, @cubicgarden & @nukeador to list just a few). I learned about the The Attitude Academy, learned I want an octocopter, talked to the folks at visionOntv, who are working hard to enable anyone to use video for social change, saw a room full of people focused on building a new generation of web & media makers and found out that I actually love popcorn(.js)!

But most importantly, @amilewski and I took Mobile Air Mozilla on the road.  We showed up at the Science Fair. We talked to people. We learned from people.  We even got talked into streaming the keynotes and closing on Air Mozilla.  And it worked.

I learned:

  • Open source video production is hard.  We both spent a lot of time trying to get the WebM capture box we had working.  This should not be this complicated.
  • The website,, was less interesting to most.
  • Most I met were more more interested in the technology we had onsite, the technology to stream from remote locations.

Some of the brain storming ideas that came out from this:

  • Create iOS/Android app to capture video and stream to Air Mozilla.  Air Mozilla will transcode and broadcast.  App could even store video for offline transfers.  Make it easy to capture and broadcast video.
  • Create web app that grabs live video and transcodes.  Why do I need to drag a special hardware capture/encoder box along?
  • How can Mozilla help push the needle on producing WebM video?

This was a great experience.  I walk away from every community event like this jazzed.  This was no exception!

Step 1.01: Mozilla IT @ MozCamp

We are just about two weeks away from MozCamp Europe (Nov 12-13) and three from MozCamp Asia (Nov 18-20)!

As I talked about in Step 1, we want to figure out how we can leverage our resources and expertise to support the entire Mozilla Community and build a better, stronger Mozilla.  At each MozCamp we’ll be hosting a workshop to talk and learn from you.

Even though we’re a couple weeks out, I wanted to share with you a draft version of the slide deck.  Take a look – Mozilla IT & Community IT.

ps. Have you joined our Mozilla Community directory @ ?