60 Seconds with Mozilla IT – April 8

mrz

1

Every Monday Mozilla holds its weekly Project Meeting.

For the past three weeks Mozilla IT has been sharing 60 Seconds with Mozilla IT.  We’ll be cross-posting those notes here.

From April 8:

2013 Q1 Review, in 3 bullet points

  • Rolled out Bugzilla upgrade from 4.0 to 4.2. And moved it to faster hardware. Shout out to Byron, David & Mark from the BMO team.
  • Moved to a new office in Paris. Monumental amount of effort to have Internet, wifi and video conferencing up in a matter of days.
  • Our product delivery CDN peaked at nearly 200Gbps. What’s a “jigga bit”? That’s like downloading a full length movie every second. Or watching 5600 Blu-ray movies AT THE SAME TIME. Or like downloading 5 DVDs every second.

We didn’t sit still last week

  • Both the Web Tools and DBA teams helped Mozilla IT proactively upgrade our PostgreSQL infrastructure, in response to major security vulnerability release, starting with Socorro crash reporting systems
  • Pushed out the first official release of the Firefox Health Report

ANNOUNCEMENT: Data Center Consolidation

mrz

4

Mozilla IT has data centers all over the world; in China, The Netherlands, Phoenix, Santa Clara and Mountain View.

Many of these locations are subject to Daylight Saving Time where the time shifts ahead an hour in the Spring and shifts back an hour in the Fall. This causes all sorts of havoc with computers and makes it very difficult to schedule meetings with such a large and geo-distributed team.  Someone‘s always having to take a meeting in the middle of the night.

In Q2, in order to provide the best service for Mozillians, Mozilla IT will be moving our entire infrastructure to one location in the South Pacific to Pacific Republic of Kiribati (UTC+14), a small island nation in Oceania.

This move will take some coordination, but we have been working with the government and Telecom Services Kiribati Limited to make sure the transition is smooth. Says Derek Moore, Mozilla’s Data Center Operations Manager, “Kiribati’s tropical location will provide a large amount of sunlight necessary for Mozilla’s new solar powered data center.”

Other than the obvious benefits of consolidation, we chose Kiribati for two reasons:

  1. Kiribati does not do any time changes for Daylight Saving Time.
  2. Kiribati is at the highest time zone. It is the first time zone to see a new day. We feel that this represents Mozilla IT’s commitment to using new technologies to brighten everyone’s day.

This small island nation sees the first light of day and Mozilla IT will literally be on the bleeding edge. As I’ve said, “the web moves fast, and Mozilla needs to move fast too.”

We expect to have this consolidation completed by 2014 April 1.

When I Moved Abroad

Sheeri

9

I grew up in the States. While growing up, I got interested in the country of Garistan. I was drawn to a country full hard-core passion for living a happy life. I studied the history and geography of Garistan, and got a pretty good feel for it.

So I moved there. At first, things seemed to go OK. Sure, there was a bit of a language barrier, but I was committed. I had a phrasebook. Some days were great, I did everything I needed to do and felt on top of the world. Some days I felt clumsy just trying to do normal tasks like food shopping. But I kept on going.

One day, I found a group of ex-patriots from the US. I went to the first meeting, a bit nervous, not quite sure what I was seeking out. What I found was relief. I could speak English without worry about how fast I was talking, or that anyone would assume I was a tourist. I could speak broken Garistanese without worrying about my accent and without worrying that I was being judged. I could talk about American things without having to explain it in detail. It was so calming, and it made those hard, clumsy days just a bit easier, because I would go back to the group and talk about struggling for a word, or my embarrassment.

We would also go out together, and there was strength in our numbers. I noticed that we all spoke better Garistanese when we were out with each other. I guess we just felt more comfortable.

Then Christmas came around. And let me tell you, it was unbelievably difficult. I love Christmas. Decorating the tree, baking cookies, shopping, the way everyone is nice to each other for 6 weeks (except when trying to find a parking space). Best of all, I love singing Christmas carols.

My first Christmas in Garistan made me realize just how different I really was. They put cotton candy on their trees to decorate them? What’s with the buckets in front of the fireplace? Tell me again what the traditional Garistan Christmas cake is? There are different songs….and even the familiar ones are sung in Garistanese.

I was out of my element. It was upsetting and frustrating. I could do what I wanted in my own apartment, but venturing forth into the street just reminded me how different I really was. My group of English-speaking US compatriots were the best gift I had that year. They helped me navigate the Christmas differences, and they even pointed out some similarities I was taking for granted. We did our own American-style traditions, including a cookie swap. It was so comfortable, and made that first Christmas really bearable.

But I cannot deny that I felt I was an outsider. I would have moved back to the States if it weren’t for my group, encouraging me and just being the same as me. It was not really the Garistan people that had me feeling so uncomfortable, it was the culture in general. It is not what I am used to, and several times I felt as though I was being treated rudely, though now that I have more experience with the Garistan community, I understand that is the way they interact, and culturally it is not thought of as rude. My Garistan friends would ask me to come out with them, but sometimes I did not want to go to a large crowded party full of all those things that made me feel uncomfortable.

SPOILER: Garistan is a made-up country. This story is an analogy about women in tech. Think of Garistan as a random tech community – maybe it’s MySQL, maybe it’s Python, maybe it’s sysadmins, devops, whatever. The group of US expatriates? That is a women-only space.

It is not sexist to have a women-only space in a community that’s so very male-oriented and has so many men in it. It is (arguably) a necessity if you want more women to be a part of the community. If you want more folks from the US to move to Garistan, wouldn’t it make sense to have comfortable spaces for those who speak English?

Christmas in this story could represent a conference or big event. I read an article that complained about women-only hackathons, because a men-only hackathon would be sexist, so of course a women-only hackathon would be sexist. And it is so utterly and completely wrong. Most men do not need a comfortable space to be among other men during a tech conference, because the conference is already mostly men. Just like most folks in Garistan do not need “Garistanese-only” spaces, because Garistanese is the dominant language there.

Is it the fault of Garistan people, that they have a Garistanese culture? No. And it is not the fault of men that they have a male culture. But if you want to retain those who come from the US, you have to change a little, offer up some more English, maybe acknowledge the Fourth of July in some little way. Same thing with tech culture – if you want to retain more women in the community, you have to make the space a little more comfortable for us.

I would also like to point out that not all women feel this uncomfortable. In fact, I actually do not feel this way most of the time. A mostly-male space does not feel that foreign to me. I always hung out with my two brothers, most of my friends growing up were male, and I actually have to put effort into being friends with other women. You can think of this as someone who moved to Garistan after visiting every year for decades. I know how “male spaces” work and I am comfortable in them.

Some women are comfortable in mostly male spaces. That does not indicate that those spaces are welcoming towards women. Maybe they are, maybe they are not. The presence of a few women means that those women are comfortable enough, but that does not mean that many women would feel comfortable there.

The level of comfort that any one woman has can vary, not just on their own experiences, but also depending on the community, and depending on her own situation. Back to the analogy, someone might be completely comfortable with the Garistanese language, having studied it, but still have a hard time with Garistanese culture. Or maybe that person is fine with everything except the different Christmas traditions. Most of the time, that person would fit right in, but Christmas is a trigger point.

If I make a “math is hard!” joke among my coworkers, they know I am smart and maybe just having an off day. If I make the same joke and someone’s listening who does not know me, they might get the impression that I really do not know what I am doing.

I will freely admit that I laugh at inappropriate humor on a locked-down IRC channel, because I know the audience, and I know the intention of the person telling the joke. Those same jokes on a public channel or at a party where I did not know anyone or even on Facebook have caused me to speak up and say “that is not funny.” Context is critical.

To the men reading this – you have a male culture. That is perfectly OK, just like Garistan has a Garistanese culture and everyone speaks Garistanese. But if you want more women in a particular space, you have to change the culture in that space. If you do not know everyone who is listening, be more thoughtful about what you say. And try to remember that we women are trying to learn technical stuff (Garistanese) and cultural stuff (Christmas traditions) at the same time.

Videos from Open Database Camp

Sheeri

3

Open Database Camp was just over a week ago, Mar 16-17th at Harvard University, co-located with Northeast LinuxFest. We had a great lineup of speakers, and we have processed all 11 videos in record time! We got new video cameras at the beginning of the year, so the video quality and resolution is stellar, you can see everything. Here are the videos:

2013 Open Database Camp
and Related Northeast LinuxFest Videos

Enjoy!

Happy Power Outage Day!

Derek Moore

2

Mozilla’s flagship datacenter, SCL3, underwent its first major stress test yesterday. Lucky for us, we had the advantage of knowing it was coming. The facility operators had scheduled significant electrical power maintenance which would result in the sequential shutdown of both our primary and secondary power feeds. We would need to absorb several power “failures” as we transferred back and forth between feeds over a 24-hour period.

fox2mike is a jerk

 

From day one, we designed our infrastructure for maximum redundancy. Every power branch is color-coded, from the PDU to the overhead bus and even right down to the equipment cords. This allows us, at a glance, to determine exactly which power supply on each server will be impacted by an electrical event anywhere in the facility. It also helps avoid operator error by providing visual confirmation when equipment is properly cabled. In the case of this maintenance, these visual cues dramatically simplified our pre-game audit and let us sleep peacefully the night before.

mozilla-power-colors

 

So, how did we do? Well, we learned a few lessons:

Cheap comes at a price

Although the vast majority of our hardware is datacenter-class, there are a few legacy servers which lack the redundant power supplies to ride out an event like this. For these servers, we rely on Automatic Transfer Switches to provide external failover on an as-needed basis. To be fully effective, this technology generally requires phase synchronization between the power branches… a complex capability not included in our design for this facility. As a result, our success rate during the transfer was about 90% (a good score on your CCIE exam, but not down in the trenches). Two devices were powercycled due to slow failover, but at least the ATS enabled an immediate, automated recovery.

Quality control matters

My biggest fear during datacenter maintenance is the human factor. Our racks are dense, and it’s surprisingly easy to knock a power cord loose while you’re shoulder-deep in copper spaghetti. We take advantage of several innovations to physically protect our cabling against drive-by disconnections. For the power cords, we use retention sleeves to ensure a more resilient mechanical connection.

mozilla-retention-sleeves

Our first batch, however, had some loose manufacturing tolerances. After application, the cord may visually appear to be fully inserted without having made an actual electrical connection. Because our equipment has multiple PSUs, the absence of a single power feed could potentially go unnoticed… until that feed is the only one left. As a result of this oversight, we lost two additional devices on the floor, and they took an outage when the redundant power was cut.


Despite these issues, we continued to serve full production load out of this facility for the duration of the maintenance. Our core, backbone, and high-density equipment functioned exactly as designed, and that instills a lot of confidence as we move forward. The incidents we did encounter helped to highlight possible shortcomings, and it only improves our design and response in the future.

 

Open Database Camp Schedule Is Up!

Sheeri

I have had a lot of folks wanting to know when talks would be during Open Database Camp, and we had enough space in the schedule that we did not need to vote on talks, so I present the Open Database Camp Boston Schedule, now online. Lots of MySQL talks, but also a Mongo talk thrown in for good measure! (still hoping to sneak a Postgres talk in…)

As with any conference schedule, this might change. See you at Harvard University in Cambridge on Saturday and Sunday!

Deprecated, Removed and Ignored Variables in MySQL 5.6

Sheeri

2

Over at the OurSQL podcast, Gerry and I were inspired by the Percona blog post about MySQL 5.5 and 5.6 default variable values differences. We were going to do a show where we talked about that, but in researching that topic, we found there were lots (around 20 to be exact) of variables and a few features in MySQL 5.6 that are ignored, removed or deprecated.

These are variables that should be removed from your configuration so as not to cause warnings or errors. When I was writing up the show notes I realized that it was a pretty good list of variables, that anyone can just read – whether or not you are willing/able to listen to the 28-minute podcast.

So if you want to see the list of variables that are deprecated, removed and ignored, complete with their workarounds/improvements, check out OurSQL Episode 130: Retired Variables.

Different MySQL Forks for Different Folks

Sheeri

5

At Confoo last week, I tried out a new presentation, called “Different MySQL Forks for Different Folks”. The idea was to explain the differences among all the forks – Drizzle, MariaDB, Percona and of course Oracle’s MySQL. But I did not just go into technical merit of each fork; I also explained the values of each company, as that can be a big decision in deciding what software to choose.

There are PDF slides and a video on youtube. But I wanted to put some of the links I used to gather information here, as an easy place for folks to come and click links if they like, or if they want to research on their own.

At the official Drizzle documentation:
What is Drizzle?
Drizzle differences compared to MySQL, including gotchas

Oracle:
MySQL Connect, Oracle’s business/technical conference for MySQL
Virtual Developer Days, March 12th and 19th

From the Percona website:
Percona Server
Percona Software
Percona 5.5 vs. MySQL 5.5 feature comparison
Percona Live, Percona’s business/technical conferences for MySQL
Why Percona Live?

About MariaDB:
MariaDB vs. MySQL compatibility
What is in the different MariaDB Releases?
Tweet asking about making MariaDB 10, MariaDB 56
What’s in MariaDB 10

Hopefully the video gives a good overview; I know that many will learn a lot from this controversial video.

Keeping IT Positively Visibly Relevant

Sheeri

1

Back in December in an IT strategy meeting, my boss came up with a great phrase – “keeping IT visibly relevant”. It sums up the long-standing problem most IT teams have in their companies – how can you make it so IT is seen as positive and good? And not just “really great when the fires happen” but day-to-day?

How can we keep IT visibly relevant, in a positive way?

Part of the problem is prioritization (IT is currently understaffed, has been for years, and our headcount for 2013 does not actually bring us up to full staffing). It’s difficult to balance the “must get done” quarterly goals with the day-to-day requests, but we also cannot say “no” all the time. Because at the end of the day, it’s not that “I can’t do this thing for you because I need to do this thing for my team,” it’s really a matter of what’s most important to Mozilla as a whole.

How do you keep IT visibly relevant, in a positive way?

I asked this on twitter and the responses ranged from “shut down printers” to “organize the office pool” to the more helpful “make it to meetings and be involved” and “present at user groups”. See:

Certainly, being forward-moving in technology is something we do. We speak at conferences, we help run http://hangops.com/ all while getting the rest of our jobs done. However, this shows that Mozilla IT is visibly relevant to those *outside* the company. And again, balance is difficult – working remotely definitely is a boon, in the last 6 weeks I presented at Linux.conf.au in Canberra (Australia), RMOUG Training Days in Denver, Scale11x in Los Angeles and Confoo in Montreal. I would have had to take a lot of time off work if I could not work remotely! But time at conferences *is* time away from the office, so there’s another point to balance there.

So, I am looking for your tips and tricks. What has worked for you?

Scheduled Maintenance – bugzilla.mozilla.org

Shyam Mani

2

A six (6) hour maintenance window is planned to upgrade bugzilla.mozilla.org on Tuesday, March 5th 2013 between 5:30pm PST (0130 UTC) to 11:30pm PST (0730 UTC)

What will be impacted
bugzilla.mozilla.org will be upgraded from version 4.0 to 4.2 and moved to newer hardware as well as an upgraded operating system. As part of the upgrade, bugzilla.mozilla.org will be relocated from Phoenix to its new home in our Santa Clara datacenter.
During this 6 hour period, bugzilla.mozilla.org will be unavailable.

What are the benefits of this upgrade
There are several features in 4.2 that improve user experience including an improved searching system and overall better performance.
Additional information on version 4.2 can be found in the release notes.

What’s next with Bugzilla
This is the first step of two in providing greater resiliency to bugzilla.mozilla.org. In Q2, IT will setup bugzilla.mozilla.org to run concurrently out of Phoenix and Santa Clara providing geo-redundancy and greater scalability.

For more information
If you have questions or would like further information or for any issues during or following the maintenance period, please leave a comment here.
More information on the maintenance policy for bugzilla.mozilla.org and other Developer Services components can be found on the maintenance page.