How to

Most operations on DNS happens in the admin/dns/domains repository (git@git-rw.torproject.org:admin/dns/domains). Those zones contains the master copy of the zone files, stored as (mostly) standard Bind zonefiles (RFC 1034), but notably without a SOA.

Tor's DNS support is fully authenticated with DNSSEC, both to the outside world but also internally, where all TPO hosts use DNSSEC in their resolvers.

Adding and editing a zone

A new zone or zone records can be added or modified to a zone in the domains git and a push. DNSSEC records are managed automatically by manage-dnssec-keys in the dns/dns-helpers git repository. through a cron job in the dnsadm user on the master DNS server (currently nevii).

Serial numbers are managed automatically by the git repository hooks.

Removing a zone

  • git grep the domain in the tor-nagios git repository
  • remove the zone in the dns/domains git repo
  • on nevii, remove the generated zonefiles and keys:

    cd /srv/dns.torproject.org/var/keys/
    mv generated/torproject.fr* OLD-generated/
    mv keys/torproject.fr OLD-KEYS/
    
  • remove the zone from the secondaries (Netnod and our own servers). this means visiting the Netnod web interface for that side, and Puppet (modules/bind/templates/named.conf.torproject-zones.erb) for our own
  • the domains will probably be listed in other locations, grep Puppet for Apache virtual hosts and email aliases
  • the domains will also probably exist in the letsencrypt-domains repository

DS records expiry and renewal

A special case is the rotation of the DNSKEY / DS records. Those rotate about once every two years, and require manual operation on the registrar (currently https://joker.com).

A Nagios hook is in /srv/dns.torproject.org/bin/dsa-check-and-extend-DS, and basically wraps manage-dnssec-keys with some Nagios status codes. It will warn when the key is about to expire and extend it before it expires (while still flagging a critical warning in Nagios).

To fix this error, you need to visit joker.com and authenticate with the password in hosts-extra-info in tor-passwords, along with the 2FA dance. Then:

  1. click on the gear next to the domain affected
  2. edit the DNSSEC section
  3. click "more" to add a record

The new key should already be present on the DNS master (currently nevii) in:

/srv/dns.torproject.org/var/keys/$DOMAIN/dsset

It is in the format (from rfc4034):

domain IN DS keytag algo type digest

For example:

torproject.net.  IN DS 53722 8 2 6d3d2be639594ffe34d4c5b9214fe5ddf81b8ee1c8505f5ec1a800dc4a809a91; Pub: 2019-05-25 17:40:08;  Act: 2019-05-25 17:40:08;  Inact: 2021-09-11 17:40:08;  Del: 2021-09-11 17:40:08;  Rev: 2021-08-12 17:40:08

With the above, you would have the following in Joker:

  • alg: 8
  • digest: 6d3d2be639594ffe34d4c5b9214fe5ddf81b8ee1c8505f5ec1a800dc4a809a91
  • type: 2
  • keytag: 53722

And click "save".

After a little while, you should be able to check if the new DS record works on DNSviz.net, for example, the DNSviz.net view of torproject.net should be sane.

The changes will take a while (~10 hours?) to trickle out into all caches, so it might take a while for the Nagios check to return green.

Eventually, Nagios will complain about the old keys, and we can remove them. Make sure to remoce the old key, not the new key. Be careful because the web interface might sort the keys in an unexpected way. check the keytag and compare with the expiration specified in the dsset file.

Note: this procedure could be automated by talking with the registrar's API, for example Joker.com's DMAPI domain modification API (see also those docs). There are also proposals at the IETF to allow delegation from the parent zone to allow the child zone to perform those updates on its own.

Further, puppet ships trust anchors for some of our zones to our unbounds. If you updated the DS for one of those, update the corresponding file in tsa-puppet/modules/unbound/files. Existing machines don't need that (since we do slow, RFC5011-style rolling of KSKs), but new instances will be sad if we ship them obsolete trust anchors.