Oct 10

Implementing DNSSEC for mozilla.org

Every quarter, Mozilla’s IT team (just like so many other teams in the organization) decides on a number of goals or tasks for that quarter. One of the goals for last quarter was implementing DNSSEC for mozilla.org. I’ve never had a chance to work hands on with DNS in a large setup…it has always been “managed” DNS and that was never much of a challenge. DNSSEC was an awesome goal to work on and I had a lot of fun working on it.

At first sight, DNSSEC is a little daunting – fairly new technology with a gazillion specs and RFCs but once you get a hang of the concepts, it’s easy to work with.

The Basics – I’m not going to spend much time here; there is a lot of information out on the internet already. If you have no idea about DNSSEC,┬áthe Wikipedia article is a great start.

The Setup – Let’s start off with how we do DNS at Mozilla :

DNS setup for Mozilla

Our DNS configs are stored in SVN. DNS config changes are committed to SVN and are then pulled down by the various DNS servers. Masters and slaves know to pull their own specific configs. All servers run a config check and restart/reload named if all is fine or drop an email if something is busted. That ensures we catch any configuration errors before they have a chance to cause any major damage.

Since the current model was working so well for us, I didn’t want to change much in terms of how the overall setup with DNSSEC operated. One of the most important aspects of DNSSEC are the keys, i.e.; the ZSKs and KSKs. The private keys have to be protected, ideally even stored offline and used only when zones or other keys have to be signed. Having the keys offline sounded a little too paranoid to start with so I just decided to go with a signer box which does the job. The new setup looks like :

DNSSEC setup for Mozilla

The signer is isolated from the internet and cannot be reached from any of the public machines. This is where we store our all our keys and this is the only machine where signing happens. The keys don’t leave the box and access is quite heavily restricted. This ensures that the chances of someone being able to manipulate our keys and hence invalidate DNSSEC are extremely minimal. Also, if for any reason a nameserver is compromised, the damage is limited as there are no keys on it.

So I want to do it too – Yay! It’s really nice and quite easy to implement DNSSEC for your domains. Here are a few steps to think about –

  1. Check if your TLD has been signed. As of writing this, .org and .info are signed for example, while .com & .net aren’t. ICANN’s TLD DNSSEC report is a good resource to check and is quite up-to-date. If your TLD isn’t signed yet, you can still go ahead but your domain will be an island of trust.
  2. Check with your registrar if they support DNSSEC. This is sort of crucial to the whole process. If your registrar doesn’t support DNSSEC, it’s going to be impossible for you to have a complete chain of trust from the root. For example, pir.org’s list of .org registrars is fairly comprehensive and even helps sort by DNSSEC compliance.
  3. Make sure your DNS software of choice supports it. We use bind at Mozilla and versions upwards of 9.3 should be good enough for DNSSEC. I’d really recommend 9.7+ though, some of the features added for DNSSEC make it really useful. If you’re on RHEL, it’s really easy to grab the RHEL 6 Beta SRPMS and rebuild them for RHEL 5.x .
  4. Your keys are crucial. dnssec-keygen is the tool of choice for bind. You’ll probably want at least 1 KSK and ZSK and it’s always nicer to have 2 of each with one on stand-by. Also, you need to think of authenticated denial of existence and if you’re going to go with NSEC or NSEC3 as this is something you’ll have to pass to dnssec-keygen when creating your keys. I’d recommend NSEC3, rfc 5155 has the gory details.
  5. Signatures have a finite lifetime. Once your domain is signed – if for some reason you lose your keys or your access to them or don’t re-sign them frequently enough, you run the risk of your signatures expiring and your whole domain being un-resolvable till the TTLs expire or your new keys get picked up various servers. While this isn’t a real concern at the moment because there aren’t a lot of validating resolvers out there… it surely is something to keep in mind for the future. Planning ahead is always a good thing!
  6. Sign your zones. dnssec-signzone will handle this for you. If you created a nice separate “key store” directory which has all your keys, the -S option to dnssec-signzone will automagically pick up the right keys and sign your zones. It’ll even work out the various times, like activation, publication and revocation times before deciding which key to use. Also, once the signing is complete, you’ll have a .signed file for your zone, like mozilla.org.signed. You’ll also find a dsset.mozilla.org. file which contains the DS records. We’ll come to why they’re important in a bit.
  7. Enable DNSSEC on your server. Adjust your configs to point to the signed domain, mozilla.org.signed. Restart named and when you run dig +dnssec mozilla.org, you should be able to see RRSIG records with your responses.
  8. At this point, you’re an island of trust. While you have DNSSEC up and running, external parties cannot verify your identity, so to speak. This is where the DS records come in. You should pass on your DS records to your registrar and ask them to update the parent zone with them. Using these DS records, resolvers can then find out DNSKEYs (public portions of your KSK and ZSK) and verify your signatures, thus completing the chain of trust.
  9. Do not push out your DS records before you have signed records ready. The moment a DS record is live, DNSSEC enabled or validating resolvers will think your domain is signed and will start looking for corresponding DNSKEY records. If they cannot find them, your domain will fail to resolve for them. This is exactly what I did and it forced us to push out DNSSEC 12 hours before it was supposed to go live and ended up breaking mozilla.org for people with validating resolvers for a while.

Once you’ve done all that, you’re good to go! Hop over to a site like dnsviz to verify you’ve got everything working fine. An example for hg.mozilla.org looks like :

hg.mozilla.org DNSSEC graph

Happy DNSSEC’ing! Next up – starting from scratch to DNSSEC ready in a few easy steps.

Oct 10

Hello World, as the standard title goes…

I’m Shyam and I’ve been working with IT @ Mozilla for over a year now. Figured I’d fire up a blog here to talk about the stuff I’ve been working on and spread some knowledge around :)

Watch this space in the near future!