Last week at PuppetConf Justin Dow and I gave a quick talk about how we scale our puppet infrastructure here at Mozilla and what we’ve learned along the way. During this learning process we saw the need for a standardized method to temporarily disable puppet. Sometimes our engineers wanted us to enable debug logs for a particular host, or sometimes you just need to make a bunch of temporary changes quickly and you’re too impatient to wait for puppet to do it for you.
The brilliant developers at Puppet Labs saw the need for this and provided us with the ability to administratively disable a node using the –disable flag, however, this wasn’t a perfect solution. At one point our nodes got into a state where puppet wouldn’t run at all, unless enabled the node immediately before running the agent. Instead of waiting for this to be fixed we just added –enable to our cron task and stopped using –disable entirely. As a result of this our operations staff all came up with their own unique way of disabling puppet. Some would comment the crontab out, some would stop crond all together, and some would chmod 0 /usr/bin/puppet.
All of those approaches have their own obvious drawbacks (stopping crond could be a really bad idea) but they all share a common flaw. You have to remember to re-enable the agent when you’re done. Periodically we would look at our dashboard and investigate the nodes which were unresponsive and quickly find that puppet had been disabled. Now the hard part began, finding out who disabled puppet and why. Our security team has all of this information has an audit trail of this, but asking them to investigate our own staff is hardly a good use of their time. It wasn’t the right solution.
We then created the tool called puppetctl, this would allow an administrator to create a window where puppet would not run. At the end of the window puppet would automatically re-enable itself. It kept track of who disabled puppet, and required you to specify a reason. It eliminated the potential for someone to forget to re-enable puppet once they were done on the system.
Download it, check it out, submit bug reports, and let us know what you think!