How to install a new bare metal server at Hetzner

This is for setting up physical metal at Hetzner.

Order

  1. get approval for the server, picking the specs from the main website

  2. head to the order page and pick the right server. pay close attention to the location, you might want to put it alongside other TPO servers (or not!) depending on redundancy or traffic requirements. Click Add to shopping cart, leaving all other fields as default.

  3. in the Server login details page, you should leave Type set to Public key. If you do not recognize your public SSH key in there, head to the server list and click on key management to add your public keys

  4. when you're certain of everything, click Checkout in the cart, review the order again and click Order in obligation.

A confirmation email will be sent by Hetzner at the TPA alias when the order is filed. Then you wait for the order to complete before being able to proceed with the install.

Ordering physical servers from Hetzner can be very fast: we've seen 2 minutes turn around times.

Install

At this point you should have received an email from Hetzner with a subject like:

Subject: Your ordered SX62 server

It should contain the SSH fingerprint, and IP address of the new host which we'll use below.

  1. login to the server using the IP address and host key hash provided above:

    ssh -o FingerprintHash=md5 -o UserKnownHostsFile=~/.ssh/authorized_keys.hetzner-rescue root@159.69.63.226
    

    Note: the FingerprintHash parameter above is to make sure we match the hashing algorithm used by Hetzner in their email, which is, at the time of writing, MD5 (!). Newer versions of SSH will also encode the hash as base64 instead of hexadecimal, so you might want to decode the base64 into the latter using this: The UserKnownHostsFile is to make sure we don't store the (temporary) SSH host key.

    perl -MMIME::Base64 -e '$h = unpack("H*", decode_base64(<>)); $h =~ s/(..)(?=.)/\1:/g; print $h, "\n"'
    
  2. Set a hostname (short version, not the FQDN):

    echo -n 'New hostname: ' && read hn && hostname "$hn" && exec bash
    

    TODO: merge this with wrapper script below.

  3. Partition disks. This might vary wildly between hosts, but in general, we want:

    • GPT partitionning, with space for a 8MB grub partition and cleartext /boot
    • software RAID (RAID-1 for two drives, RAID-5 for 3, RAID-10 for 4)
    • crypto (LUKS)
    • LVM, with separate volume groups for different medium (SSD vs HDD)

    We are experimenting with FAI's setup-storage to partition disks instead of rolling our own scripts. You first need to checkout the installer's configuration:

        apt install git
        git clone https://git.torproject.org/admin/tsa-misc
        cd tsa-misc/installer
        git show-ref master
    

    Check that the above hashes match a trusted copy.

    Use the following to setup a Ganeti node, for example:

        apt install fai-setup-storage
    
        setup-storage -f "disk-config/gnt-fsn" -X
    

    TODO: merge this with wrapper script below.

    TODO: convert the other existing tor-install-format-disks-4HDDs script into a setup-storage configuration.

    And finally mount the filesystems:

        . /tmp/fai/disk_var.sh &&
        mkdir /target &&
        mount "$ROOT_PARTITION" /target &&
        mkdir /target/boot &&
        mount "$BOOT_DEVICE" /target/boot
    

    TODO: test if we can skip that test by passing $ROOT_PARTITION as a --target to grml-debootstrap. Probably not.

    TODO: in any case, this could be all wrapper up in a single wrapper shell script in tsa-misc instead of this long copy-paste. Possibly merge with tor-install-hetzner from new-machine-hetzner-cloud.

  4. Install the system. This can be done with grml-debootstrap which will also configure grub, a root password and so on. This should get you started, assuming the formatted root disk is mounted on /target and that the boot device is defined by $BOOT_DEVICE (populated above by FAI). Note that BOOT_DISK is the disk as opposed to the partition which is $BOOT_DEVICE.

    BOOT_DISK=/dev/nvme0n1 &&
    mkdir -p /target/run && mount -t tmpfs tgt-run /target/run &&
    mkdir /target/run/udev && mount -o bind /run/udev /target/run/udev &&
    apt-get install -y grml-debootstrap && \
    grml-debootstrap \
        --grub "$BOOT_DISK" \
        --target /target \
        --hostname `hostname` \
        --release buster \
        --mirror https://mirror.hetzner.de/debian/packages/ \
        --packages /root/tsa-misc/installer/packages \
        --post-scripts /root/tsa-misc/installer/post-scripts/ \
        --nopassword \
        --remove-configs \
        --defaultinterfaces &&
    umount /target/run/udev /target/run
    
  5. setup dropbear-initramfs to unlock the filesystem on boot. this should already have been done by the 50-tor-install-luks-setup hook deployed in the grml-debootstrap stage.

    TODO: in an install following the above procedure, a keyfile was left unprotected in /etc. Make sure we have strong mechanisms to avoid that ever happening again. For example:

    chmod 0 /etc/luks/
    

    TODO: the keyfiles deployed there can be used to bootstrap mandos. Document how to do this better.

  6. Review the crypto configuration:

    cat /target/etc/crypttab
    

    If the backing device is NOT an SSD, remove the ,discard option.

    TODO: remove this step, it is probably unnecessary.

  7. Review the network configuration, since it will end up in the installed instance:

    cat /target/etc/network/interfaces
    

    An example safe configuration is:

    auto lo
    iface lo inet loopback
    
    allow-hotplug eth0
    iface eth0 inet dhcp
    

    The latter two lines usually need to be added as they are missing from Hetzner rescue shells:

    cat >> /etc/network/interfaces <<EOF
    
    allow-hotplug eth0
    iface eth0 inet dhcp
    EOF
    

    TODO: fix this in a post-install debootstrap hook, or in grml-debootstrap already, see also upstream issue 105 and issue 136.

    Add the hostname, IP address and domain to /etc/hosts and /etc/resolv.conf:

    grep torproject.org /etc/resolv.conf || ( echo 'domain torproject.org'; echo 'nameserver 8.8.8.8' ) >> /etc/resolv.conf
    if ! hostname -f 2>/dev/null || [ "$(hostname)" = "$(hostname -f)" ]; then
        IPADDRESS=$(ip -br -color=never route get to 8.8.8.8 | head -1 | grep -v linkdown | sed 's/.*  *src  *\([^ ]*\)  *.*/\1/')
        echo "$IPADDRESS $(hostname).torproject.org $(hostname)" >> /etc/hosts
    fi
    

    TODO: add the above as a post-hook. possibly merge with tor-puppet/3rdparty/modules/ganeti/files/instance-debootstrap/hooks/gnt-debian-interfaces

    TODO: add IPv6 address configuration. look at how tor-install-generate-ldap guesses as well.

  8. If any of those latter things changed, you need to regenerate the initramfs:

    chroot /target update-initramfs -u
    chroot /target update-grub
    

    TODO: remove this step, if the above extra steps are removed.

  9. umount things:

    umount /target/run/udev || true &&
    for fs in dev proc run sys  ; do
        umount /target/$fs || true
    done &&
    umount /target/boot &&
    cd / && umount /target
    

    TODO: merge this with wrapper script.

  10. close things

    vgchange -a n cryptsetup luksClose crypt_dev_md1 cryptsetup luksClose crypt_dev_md2 mdadm --stop /dev/md*

    TODO: merge this with wrapper script.

  11. Document the LUKS passphrase and root password in tor-passwords

  12. Cross fingers and reboot:

    reboot

Configuration

See new-machine for post-install configuration steps, then follow new-machine-mandos for setting up the mandos client on this host.