How to install a new bare metal server at Hetzner

This is for setting up physical metal at Hetzner.

Order

  1. get approval for the server, picking the specs from the main website

  2. head to the order page and pick the right server. pay close attention to the location, you might want to put it alongside other TPO servers (or not!) depending on redundancy or traffic requirements. Click Add to shopping cart, leaving all other fields as default.

  3. in the Server login details page, you should leave Type set to Public key. If you do not recognize your public SSH key in there, head to the server list and click on key management to add your public keys

  4. when you're certain of everything, click Checkout in the cart, review the order again and click Order in obligation.

A confirmation email will be sent by Hetzner at the TPA alias when the order is filed. Then you wait for the order to complete before being able to proceed with the install.

Ordering physical servers from Hetzner can be very fast: we've seen 2 minutes turn around times.

Install

At this point you should have received an email from Hetzner with a subject like:

Subject: Your ordered SX62 server

It should contain the SSH fingerprint, and IP address of the new host which we'll use below.

  1. login to the server using the IP address and host key hash provided above:

    ssh -o FingerprintHash=md5 -o UserKnownHostsFile=/dev/null root@159.69.63.226
    

    Note: the FingerprintHash parameter above is to make sure we match the hashing algorithm used by Hetzner in their email, which is, at the time of writing, MD5 (!). Newer versions of SSH will also encode the hash as base64 instead of hexadecimal, so you might want to decode the base64 into the latter using this: The UserKnownHostsFile is to make sure we don't store the (temporary) SSH host key.

    perl -MMIME::Base64 -e '$h = unpack("H*", decode_base64(<>)); $h =~ s/(..)(?=.)/\1:/g; print $h, "\n"'
    
  2. Partition disks. This might vary wildly between hosts, but in general, we want:

    • GPT partitionning, with space for a 8MB grub partition and cleartext /boot
    • software RAID (RAID-1 for two drives, RAID-5 for 3, RAID-10 for 4)
    • crypto (LUKS)
    • LVM, with separate volume groups for different medium (SSD vs HDD)

    This can be done with the tor-install-format-disks in the tsa-misc repository, which should be carefully checked and configured before running.

    To get the scripts onto the host, you can clone them using git clone https://git.torproject.org/admin/tsa-misc.

    Check that the master hash matches what you expect (cd tsa-misc && git show-ref master).

  3. Install the system. This can be done with grml-debootstrap which will also configure grub, a root password and so on. This should get you started, assuming the formatted root disk is mounted on /target:

    # make target/run stay clean
    mkdir /target/run && mount -t tmpfs tgt-run /target/run
    # grml-debootstrap hangs for weasel in vgs without this:
    mkdir /target/run/udev && mount -o bind /run/udev /target/run/udev
    
    ROOTPASSWORD=\$(tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 30) \
    echo -n "boot disk device: " && read bootdisk &&
    apt-get install -y grml-debootstrap && \
    sed -e 's/postfix//;
            s/vlan//;
            s/bridge-utils//;
            s/ifenslave//;
            s/resolvconf//;
            s/zsh//;
            s/strace//;
            s/os-prober//;
            s/bzip2//;
            s/file//;
            s/lsof//;
            s/most//;
            $adbus
            $acryptsetup-initramfs
            ' /etc/debootstrap/packages > /root/grml-packages &&
    grml-debootstrap --grub "$bootdisk" --target /target \
        --hostname `hostname` --release buster \
        --mirror https://mirror.hetzner.de/debian/packages/ \
        --packages /root/grml-packages \
        --password "$ROOTPASSWORD" \
        --remove-configs --defaultinterfaces
    
    umount /target/run/udev /target/run
    
  4. Once the bootstrapping is complete, you still need to make sure the system can boot as, the above doesn't (unfortunately) configure everything for you. First, make a reasonable etc/fstab:

    sed -e 's/^<span class="createlink">:space:</span>*//' > /target/etc/fstab << EOF
    /dev/$vg/root / ext4 errors=remount-ro 0 1
    /dev/md/boot /boot ext4 defaults 0 2
    
    /dev/$vg/swap none swap sw 0 0
    tmpfs /tmp tmpfs defaults,size=512m 0 0
    EOF
    
  5. setup dropbear-initramfs to unlock the filesystem on boot. this can be done with the tor-install-luks-setup in the tsa-misc repository.

    cd /target && bash /root/tsa-misc/installer/tor-install-luks-setup
    
  6. Review the crypto configuration:

    cat /target/etc/crypttab
    

    If the backing device is an SSD, add ,discard after the luks option.

  7. mount the helper filesystems once more

    for fs in dev proc run sys  ; do
      mount -o bind /$fs "/target/$fs";
    done
    
  8. Do the same with the RAID configuration, probably with something like:

    chroot /target sh -c "/usr/share/mdadm/mkconf > /etc/mdadm/mdadm.conf"
    
  9. install grub on any secondary disk, for instance

    chroot /target grub-install /dev/nvme1n1
    
  10. Review the network configuration:

    cat /target/etc/network/interfaces

    An example safe configuration is:

    iface lo inet loopback
    
    allow-hotplug eth0
    iface eth0 inet dhcp
    
  11. Copy paste your key into the root's authorized keys, just to make sure you can login:

    mkdir -p /target/root/.ssh/ &&
    cp /root/.ssh/authorized_keys /target/root/.ssh/authorized_keys
    
  12. If any of those latter things changed, you need to regenerate the initramfs:

    chroot /target update-initramfs -u
    chroot /target update-grub
    
  13. umount things umount /target/run/udev for fs in dev proc run sys ; do umount /target/$fs done umount /target/boot cd / && umount /target

  14. close things

    vgchange -a n cryptsetup luksClose cpv_nvme for i in /dev/md/*; do mdadm --stop $i; done

  15. Document the LUKS passphrase and root password in tor-passwords

  16. Cross fingers and reboot:

    reboot

Configuration

See new-machine for post-install configuration steps, then follow new-machine-mandos for setting up the mandos client on this host.