Last week in IT: Puppet Training

cshields

5

Here at Mozilla, we have grown to thousands of servers in a short period of time (and even more individual instances when you count virtual machines). Like most other organizations, we have to rely on tools that help the sysadmins keep their sanity at such scale.  We have picked Puppet as our tool of choice, and are still in the process of migrating older systems into our centralized management while making sure that future servers are built out “in puppet” first before they reach production.  For those of you unfamiliar with Puppet and similar management tools (like cfengine or chef), the idea is simple: you define the way a server should be configured and Puppet will make sure that it is always setup that way.  For instance, in not so simple terms you tell it that apache should be installed with certain vhost configurations, and needs to be restarted after those configurations are put into place.  Puppet makes that happen.  Multiply that action by hundreds of webservers and Puppet shows its value.

A puppet. Courtesy of my kids.

A puppet. Courtesy of my kids.

Last week, most of the IT team spent a few days locked in the Holodeck conference room for training offered up by Puppet Labs.  Between IT and Release Engineering we filled the room with 25 people.

So what now?  We still have the daunting task of integrating our existing infrastructure and old servers into puppet.  This kind of work is easy to put on the backburner with the other projects and bugs at hand, but in the future this work pays off.  Some day those servers will need to be retired, outgrow the current hardware capacity, or need to be moved to a new data center.  Being able to define a server state with puppet makes all of these tasks a lot easier.  More work up front pays off in the long run.

The training brought unfamiliar admins up to speed with Puppet and gave us all new ideas for refactoring some of our current modules and manifests.  We’ll be working hard throughout the coming months to make those changes.  The end result is simple, everything in our infrastructure will be managed by Puppet.  Email, webservers, collaboration tools, development resources.  If it is on our network it needs to be centrally managed. We are making great progress toward this goal and it will lead to great payoffs in the future of Mozilla’s IT infrastructure.

That’s all for this week.  Next week: we resolve a bug!

5 responses

  1. Preed wrote on ::

    We’ve been looking at using Puppet too.

    While it looks cool, the main thing I haven’t been able to get over is that Puppet describes its Win32 support as “basic”; has there been any movement on this in recent months, such that it can be used to manage Win32 hosts in production? Is Mozilla planning to do that for any Win32 hosts?

    (I know most of the server infrastructure there is probably Linux, and I’ve heard great things about it in this environment.)

  2. cshields wrote on :

    @Preed,

    Yeah I think “basic” is being a little too generous with regard to Win32 support right now. However, they are putting work into it every day and so it will only be a matter of time.

    Are we “planning” on using it for Win32 hosts? I believe we are (I can’t speak for Releng directly but I think that is the case). However, until there is some kind of package management support it doesn’t provide much for us yet.

    You are right though, at least on my side of the data center there is not much windows to manage, but we still have plenty of it on the build side.

  3. zob wrote on :

    if its open source, it can be fixed.

  4. Preed wrote on ::

    Hey Corey,

    Thanks for the answers!

    And, just in case my memory is wrong, Mozilla is currently using OPSI to manage Win32 hosts?

  5. cshields wrote on :

    For now, yes.. Puppet would be the more ideal solution though, once it supports Win32.