Ganeti is software designed to facilitate the management of virtual machines (KVM or Xen). It helps you move virtual machine instances from one node to another, create an instance with DRBD replication on another node and do the live migration from one to another, etc.

Tutorial

Listing virtual machines (instances)

This will show the running guests, known as "instances":

gnt-instance list

Accessing serial console

Our instances do serial console, starting in grub. To access it, run

gnt-instance console test01.torproject.org

To exit, use ^] -- that is, Control-<Closing Bracket>.

How-to

Glossary

In Ganeti, a physical machine is called a node and a virtual machine is an instance. A node is elected to be the master where all commands should be ran from. Nodes are interconnected through a private network that is used to communicate commands and synchronise disks (with drbd). Instances are normally assigned two nodes: a primary and a secondary: the primary is where the virtual machine actually runs and th secondary acts as a hot failover.

See also the more extensive glossary in the Ganeti documentation.

Adding a new instance

This command creates a new guest, or "instance" in Ganeti's vocabulary:

gnt-instance add \
  -o debootstrap+buster \
  -t drbd --no-wait-for-sync \
  --net 0:ip=pool,network=gnt-fsn \
  --no-ip-check \
  --no-name-check \
  --disk 0:size=10G \
  --disk 1:size=2G,name=swap \
  --disk 2:size=20G \
  --disk 3:size=800G,vg=vg_ganeti_hdd \
  --backend-parameters memory=8g,vcpus=2 \
  test-01.torproject.org

WARNING: there is a bug in ganeti-instance-debootstrap which misconfigures ping (among other things), see bug #31781. It's currently patched in our version of the Debian package, but that patch might disappear if Debian upgrade the package without shipping our patch.

This configures the following:

  • redundant disks in a DRBD mirror, use -t plain instead of -t drbd for tests as that avoids syncing of disks and will speed things up considerably (even with --no-wait-for-sync there are some operations that block on synced mirrors). Only one node should be provided as the argument for --node then.
  • three partitions: one on the default VG (SSD), one on another (HDD) and a swap file on the default VG, if you don't specify a swap device, a 512MB swapfile is created in /swapfile. TODO: configure disk 2 and 3 automatically in installer. (/var and /srv?)
  • 8GB of RAM with 2 virtual CPUs
  • an IP allocated from the public gnt-fsn pool: gnt-instance add will print the IPv4 address it picked to stdout. The IPv6 address can be found in /var/log/ganeti/os/ on the primary node of the instance, see below.
  • with the test-01.torproject.org hostname

To find the root password, ssh host key fingerprints, and the IPv6 address, run this on the node where the instance was created, for example:

egrep 'root password|configured eth0 with|SHA256' $(ls -tr /var/log/ganeti/os/* | tail -1) | grep -v $(hostname)

Note that you need to use the --node parameter to pick on which machines you want the machine to end up, otherwise Ganeti will choose for you. Use, for example, --node fsn-node-01:fsn-node-02 to use node-01 as primary and node-02 as secondary. It might be better to let the Ganeti allocator do its job since it will, eventually do this during cluster rebalancing.

We copy root's authorized keys into the new instance, so you should be able to log in with your token. You will be required to change the root password immediately. Pick something nice and document it in tor-passwords.

Also set reverse DNS for both IPv4 and IPv6 in hetzner's robot. (Chek under servers -> vSwitch -> IPs)

Then follow new-machine.

Modifying an instance

It's possible to change the IP, CPU, or memory allocation of an instance using the gnt-instance modify command:

gnt-instance modify -B vcpus=2 test1.torproject.org
gnt-instance modify -B memory=4g test1.torproject.org
gnt-instance reboot test1.torproject.org

IP address changes require a full stop and will require manual changes to the /etc/network/interfaces* files:

gnt-instance modify --net 0:modify,ip=116.202.120.175 test1.torproject.org
gnt-instance stop test1.torproject.org
gnt-instance start test1.torproject.org
gnt-instance console test1.torproject.org

The gnt-instance grow-disk command can be used to change the size of the underlying device:

gnt-instance grow-disk test1.torproject.org 0 16g
gnt-instance reboot test1.torproject.org

The number 0 in this context, indicates the first disk of the instance. Then the filesystem needs to be resized inside the VM:

ssh root@test1.torproject.org resize2fs /dev/sda1

Destroying an instance

This totally deletes the instance, including all mirrors and everything, be very careful with it:

gnt-instance remove test01.torproject.org

Disk operations (DRBD)

Instances should be setup using the DRBD backend, in which case you should probably take a look at drbd if you have problems with that. Ganeti handles most of the logic there so that should generally not be necessary.

Evaluating cluster capacity

This will list instances repeatedly, but also show their assigned memory, and compare it with the node's capacity:

watch -n5 -d 'gnt-instance list -o pnode,name,be/vcpus,be/memory,status,disk_template  |  sort; echo; gnt-node list'

The gnt-cluster verify command will also check to see if there's enough space on secondaries to account for the failure of a node. Healthy output looks like this:

root@fsn-node-01:~# gnt-cluster verify
Submitted jobs 48030, 48031
Waiting for job 48030 ...
Fri Jan 17 20:05:42 2020 * Verifying cluster config
Fri Jan 17 20:05:42 2020 * Verifying cluster certificate files
Fri Jan 17 20:05:42 2020 * Verifying hypervisor parameters
Fri Jan 17 20:05:42 2020 * Verifying all nodes belong to an existing group
Waiting for job 48031 ...
Fri Jan 17 20:05:42 2020 * Verifying group 'default'
Fri Jan 17 20:05:42 2020 * Gathering data (2 nodes)
Fri Jan 17 20:05:42 2020 * Gathering information about nodes (2 nodes)
Fri Jan 17 20:05:45 2020 * Gathering disk information (2 nodes)
Fri Jan 17 20:05:45 2020 * Verifying configuration file consistency
Fri Jan 17 20:05:45 2020 * Verifying node status
Fri Jan 17 20:05:45 2020 * Verifying instance status
Fri Jan 17 20:05:45 2020 * Verifying orphan volumes
Fri Jan 17 20:05:45 2020 * Verifying N+1 Memory redundancy
Fri Jan 17 20:05:45 2020 * Other Notes
Fri Jan 17 20:05:45 2020 * Hooks Results

A sick node would have said something like this instead:

Mon Oct 26 18:59:37 2009 * Verifying N+1 Memory redundancy
Mon Oct 26 18:59:37 2009   - ERROR: node node2: not enough memory to accommodate instance failovers should node node1 fail

See the ganeti manual for a more extensive example

Moving instances and failover

Ganeti is smart about assigning instances to nodes. There's also a command (hbal) to automatically rebalance the cluster (see below). If for some reason hbal doesn’t do what you want or you need to move things around for other reasons, here are a few commands that might be handy.

Make an instance switch to using it's secondary:

gnt-instance migrate test1.torproject.org

Make all instances on a node switch to their secondaries:

gnt-node migrate test1.torproject.org

The migrate commands does a "live" migrate which should avoid any downtime during the migration. It might be preferable to actually shutdown the machine for some reason (for example if we actually want to reboot because of a security upgrade). Or we might not be able to live-migrate because the node is down. In this case, we do a failover

gnt-instance failover test1.torproject.org

The gnt-node evacuate command can also be used to "empty" a given node altogether, in case of an emergency:

gnt-node evacuate -I . fsn-node-02.torproject.org

Similarly, the gnt-node failover command can be used to hard-recover from a completely crashed node:

gnt-node failover fsn-node-02.torproject.org

Note that you might need the --ignore-consistency flag if the node is unresponsive.

Importing external instances

Assumptions:

  • INSTANCE: name of the instance being migrated, the "old" one being outside the cluster and the "new" one being the one created inside the cluster (e.g. chiwui.torproject.org)
  • SPARE_NODE: a ganeti node with free space (e.g. fsn-node-03.torproject.org) where the INSTANCE will be migrated
  • MASTER_NODE: the master ganeti node (e.g. fsn-node-01.torproject.org)
  • KVM_HOST: the machine which we migrate the INSTANCE from
  • the INSTANCE has only root and swap partitions

Import procedure:

  1. pick a viable SPARE NODE to import the instance (see "evaluating cluster capacity" above, when in doubt), login to the three servers, setting the proper environment everywhere, for example:

    MASTER_NODE=fsn-node-01.torproject.org
    SPARE_NODE=fsn-node-03.torproject.org
    KVM_HOST=kvm1.torproject.org
    INSTANCE=test.torproject.org
    
  2. establish VM specs, on the KVM HOST:

    • disk space in GiB:

      for disk in /srv/vmstore/$INSTANCE/*; do
          printf "$disk: "
          echo "$(qemu-img info --output=json $disk | jq '."virtual-size"') / 1024 / 1024 / 1024" | bc -l
      done
      
    • number of CPU cores:

      sed -n '/<vcpu/{s/[^>]*>//;s/<.*//;p}' < /etc/libvirt/qemu/$INSTANCE.xml
      
    • memory, assuming from KiB to GiB:

      echo "$(sed -n '/<memory/{s/[^>]*>//;s/<.*//;p}' < /etc/libvirt/qemu/$INSTANCE.xml) /1024 /1024" | bc -l
      

      TODO: make sure the memory line is in KiB and that the number makes sense.

    • on the INSTANCE, find the swap device UUID so we can recreate it later:

      blkid -t TYPE=swap -s UUID -o value
      
  3. setup a copy channel, on the SPARE NODE:

    ssh-agent bash
    ssh-add /etc/ssh/ssh_host_ed25519_key
    cat /etc/ssh/ssh_host_ed25519_key.pub
    

    on the KVM HOST:

    echo "$KEY_FROM_SPARE_NODE" >> /etc/ssh/userkeys/root
    
  4. copy the .qcow file(s) over, from the KVM HOST to the SPARE NODE:

    rsync -P $KVM_HOST:/srv/vmstore/$INSTANCE/$INSTANCE-root /srv/
    rsync -P $KVM_HOST:/srv/vmstore/$INSTANCE/$INSTANCE-lvm /srv/ || true
    

    Note: it's possible there is not enough room in /srv: in the base Ganeti installs, everything is in the same root partition (/) which will fill up if the instance is (say) over ~30GiB. In that case, create a filesystem in /srv:

    (mkdir /root/srv && mv /srv/* /root/srv true) || true &&
    lvcreate -L 200G vg_ganeti -n srv &&
    mkfs /dev/vg_ganeti/srv &&
    echo "/dev/vg_ganeti/srv /srv ext4 rw,noatime,errors=remount-ro 0 2" >> /etc/fstab &&
    mount /srv &&
    ( mv /root/srv/* ; rmdir /root/srv )
    

    This partition can be reclaimed once the VM migrations are completed, as it needlessly takes up space on the node.

  5. on the SPARE NODE, create and initialize a logical volume with the predetermined size:

    lvcreate -L 4GiB -n $INSTANCE-swap vg_ganeti
    mkswap --uuid $SWAP_UUID /dev/vg_ganeti/$INSTANCE-swap
    lvcreate -L 20GiB -n $INSTANCE-root vg_ganeti
    qemu-img convert /srv/$INSTANCE-root  -O raw /dev/vg_ganeti/$INSTANCE-root
    lvcreate -L 40GiB -n $INSTANCE-lvm vg_ganeti_hdd
    qemu-img convert /srv/$INSTANCE-lvm  -O raw /dev/vg_ganeti_hdd/$INSTANCE-lvm
    

    Note how we assume two disks above, but the instance might have a different configuration that would require changing the above. The above, common, configuration is to have an LVM disk separate from the "root" disk, the former being on a HDD, but the HDD is sometimes completely omitted and sizes can differ.

    Sometimes it might be worth using pv to get progress on long transfers:

    qemu-img convert /srv/$INSTANCE-lvm -O raw /srv/$INSTANCE-lvm.raw
    pv /srv/$INSTANCE-lvm.raw | dd of=/dev/vg_ganeti_hdd/$INSTANCE-lvm bs=4k
    

    TODO: ideally, the above procedure (and many steps below as well) would be automatically deduced from the disk listing established in the first step.

  6. on the MASTER NODE, create the instance, adopting the LV:

    gnt-instance add -t plain \
        -n fsn-node-03 \
        --disk 0:adopt=$INSTANCE-root \
        --disk 1:adopt=$INSTANCE-swap \
        --disk 2:adopt=$INSTANCE-lvm,vg=vg_ganeti_hdd \
        --backend-parameters memory=2g,vcpus=2 \
        --net 0:ip=pool,network=gnt-fsn \
        --no-name-check \
        --no-ip-check \
        -o debootstrap+default \
        $INSTANCE
    
  7. cross your fingers and watch the party:

    gnt-instance console $INSTANCE
    
  8. IP address change on new instance:

    edit /etc/hosts and /etc/network/interfaces by hand and add IPv4 and IPv6 ip. IPv4 configuration can be found in:

      gnt-instance show $INSTANCE
    

    Latter can be guessed by concatenating 2a01:4f8:fff0:4f:: and the IPv6 local local address without fe80::. For example: a link local address of fe80::266:37ff:fe65:870f/64 should yield the following configuration:

      iface eth0 inet6 static
          accept_ra 0
          address 2a01:4f8:fff0:4f:266:37ff:fe65:870f/64
          gateway 2a01:4f8:fff0:4f::1
    

    TODO: reuse gnt-debian-interfaces from the ganeti puppet module script here?

  9. functional tests: change your /etc/hosts to point to the new server and see if everything still kind of works

  10. shutdown original instance

  11. resync and reconvert image, on the Ganeti MASTER NODE:

    gnt-instance stop $INSTANCE
    

    on the Ganeti node:

    rsync -P $KVM_HOST:/srv/vmstore/$INSTANCE/$INSTANCE-root /srv/ &&
     qemu-img convert /srv/$INSTANCE-root  -O raw /dev/vg_ganeti/$INSTANCE-root &&
     rsync -P $KVM_HOST:/srv/vmstore/$INSTANCE/$INSTANCE-lvm /srv/ &&
     qemu-img convert /srv/$INSTANCE-lvm  -O raw /dev/vg_ganeti_hdd/$INSTANCE-lvm
    
  12. switch to DRBD, still on the Ganeti MASTER NODE:

    gnt-instance modify -t drbd $INSTANCE
     gnt-instance failover $INSTANCE
     gnt-instance startup $INSTANCE
    
  13. redo IP adress change in /etc/network/interfaces and /etc/hosts

  14. final functional test

  15. change IP address in the following locations:

    • nagios (don't forget to change the parent)
    • LDAP (ipHostNumber field, but also change the physicalHost and l fields!)
    • Puppet (grep in tor-puppet source, run puppet agent -t; ud-replicate on pauli)
    • DNS (grep in tor-dns source, puppet agent -t; ud-replicate on nevii)
    • reverse DNS (upstream web UI, e.g. Hetzner Robot)
  16. decomission old instance (retire-a-host)

Troubleshooting

  • if boot takes a long time and you see a message like this on the console:

     [  *** ] A start job is running for dev-disk-by\x2duuid-484b5...26s / 1min 30s)
    

    ... which is generally followed by:

     [DEPEND] Dependency failed for /dev/disk/by-…6f4b5-f334-4173-8491-9353d4f94e04.
     [DEPEND] Dependency failed for Swap.
    

    it means the swap device UUID wasn't setup properly, and does not match the one provided in /etc/fstab. That is probably because you missed the mkswap -U step documented above.

References

  • Upstream docs have the canonical incantation:

     gnt-instance add -t plain -n HOME_NODE ... --disk 0:adopt=lv_name[,vg=vg_name] INSTANCE_NAME
    
  • DSA docs also use disk adoption and have a procedure to migrate to DRBD

  • Riseup docs suggest creating a VM without installing, shutting down and then syncing

Ganeti supports importing and exporting from the Open Virtualization Format (OVF), but unfortunately it doesn't seem libvirt supports exporting to OVF. There's a virt-convert tool which can import OVF, but not the reverse. The libguestfs library also has a converter but it also doesn't support exporting to OVF or anything Ganeti can load directly.

So people have written their own conversion tools or their own conversion procedure.

Ganeti also supports file-backed instances but "adoption" is specifically designed for logical volumes, so it doesn't work for our use case.

Rebooting

Those hosts need special care, as we can accomplish zero-downtime reboots on those machines. There's a script (ganeti-reboot-cluster) deployed in the ganeti cluster that can be ran on the master to migrate all instances around and perform a clean reboot.

Such a reboot should be ran interactively, inside a tmux or screen session, and takes over 15 minutes to complete right now, but depends on the size of the cluster (in terms of core memory usage).

Once the reboot is completed, all instances might end up on a single machine, and the cluster might need to be rebalanced. This is automatically scheduled by the ganeti-reboot-cluster script and will be done within 30 minutes of the reboot.

Rebalancing a cluster

After a reboot or a downtime, all nodes might end up on the same machine. This is normally handled by the reboot script, but it might be desirable to do this by hand if there was a crash or another special condition.

This can be easily corrected with this command, which will spread instances around the cluster to balance it:

hbal -L -C -v -X

This will automatically move the instances around and rebalance the cluster. Here's an example run on a small cluster:

root@fsn-node-01:~# gnt-instance list
Instance                          Hypervisor OS                 Primary_node               Status  Memory
loghost01.torproject.org          kvm        debootstrap+buster fsn-node-02.torproject.org running   2.0G
onionoo-backend-01.torproject.org kvm        debootstrap+buster fsn-node-02.torproject.org running  12.0G
static-master-fsn.torproject.org  kvm        debootstrap+buster fsn-node-02.torproject.org running   8.0G
web-fsn-01.torproject.org         kvm        debootstrap+buster fsn-node-02.torproject.org running   4.0G
web-fsn-02.torproject.org         kvm        debootstrap+buster fsn-node-02.torproject.org running   4.0G
root@fsn-node-01:~# hbal -L -X
Loaded 2 nodes, 5 instances
Group size 2 nodes, 5 instances
Selected node group: default
Initial check done: 0 bad nodes, 0 bad instances.
Initial score: 8.45007519
Trying to minimize the CV...
    1. onionoo-backend-01 fsn-node-02:fsn-node-01 => fsn-node-01:fsn-node-02   4.98124611 a=f
    2. loghost01          fsn-node-02:fsn-node-01 => fsn-node-01:fsn-node-02   1.78271883 a=f
Cluster score improved from 8.45007519 to 1.78271883
Solution length=2
Got job IDs 16345
Got job IDs 16346
root@fsn-node-01:~# gnt-instance list
Instance                          Hypervisor OS                 Primary_node               Status  Memory
loghost01.torproject.org          kvm        debootstrap+buster fsn-node-01.torproject.org running   2.0G
onionoo-backend-01.torproject.org kvm        debootstrap+buster fsn-node-01.torproject.org running  12.0G
static-master-fsn.torproject.org  kvm        debootstrap+buster fsn-node-02.torproject.org running   8.0G
web-fsn-01.torproject.org         kvm        debootstrap+buster fsn-node-02.torproject.org running   4.0G
web-fsn-02.torproject.org         kvm        debootstrap+buster fsn-node-02.torproject.org running   4.0G

In the above example, you should notice that the web-fsn instances both ended up on the same node. That's because the balancer did not know that they should be distributed. A special configuration was done, below, to avoid that problem in the future. But as a workaround, instances can also be moved by hand and the cluster re-balanced.

Redundant instances distribution

Some instances are redundant across the cluster and should not end up on the same node. A good example are the web-fsn-01 and web-fsn-02 instances which, in theory, would serve similar traffic. If they end up on the same node, it might flood the network on that machine or at least defeats the purpose of having redundant machines.

The way to ensure they get distributed properly by the balancing algorithm is to "tag" them. For the web nodes, for example, this was performed on the master:

gnt-instance add-tags web-fsn-01.torproject.org web-fsn
gnt-instance add-tags web-fsn-02.torproject.org web-fsn
gnt-cluster add-tags htools:iextags:web-fsn

This tells Ganeti that web-fsn is an "exclusion tag" and the optimizer will not try to schedule instances with those tags on the same node.

To see which tags are present, use:

# gnt-cluster list-tags
htools:iextags:web-fsn

You can also find which nodes are assigned to a tag with:

# gnt-cluster search-tags web-fsn
/cluster htools:iextags:web-fsn
/instances/web-fsn-01.torproject.org web-fsn
/instances/web-fsn-02.torproject.org web-fsn

Adding and removing addresses on instances

Say you created an instance but forgot to need to assign an extra IP. You can still do so with:

gnt-instance modify --net -1:add,ip=116.202.120.174,network=gnt-fsn test01.torproject.org

Pager playbook

I/O overload

In case of excessive I/O, it might be worth looking into which machine is in cause. The drbd page explains how to map a DRBD device to a VM. You can also find which logical volume is backing an instance (and vice versa) with this command:

lvs -o+tags

This will list all logical volumes and their associated tags. If you already know which logical volume you're looking for, you can address it directly:

root@fsn-node-01:~# lvs -o tags /dev/vg_ganeti_hdd/4091b668-1177-41ac-9310-1eac45b46620.disk2_data
  LV Tags
  originstname+bacula-director-01.torproject.org

Node failures

Ganeti clusters are designed to be self-healing. As long as only one machine disappears, the cluster should be able to recover by failing over other nodes. This is currently done manually, see the migrate section above.

This could eventually be automated if such situations occur more often, by scheduling a harep cron job, which isn't enabled in Debian by default. See also the autorepair section of the admin manual.

Other troubleshooting

Riseup has documentation on various failure scenarios including master failover, which we haven't tested. There's also upstream documentation on changing node roles which might be useful for a master failover scenario.

The walkthrough also has a few recipes to resolve common problems.

Disaster recovery

If things get completely out of hand and the cluster becomes too unreliable for service, the only solution is to rebuild another one elsewhere. Since Ganeti 2.2, there is a move-instance command to move instances between cluster that can be used for that purpose.

If Ganeti is completely destroyed and its APIs don't work anymore, the last resort is to restore all virtual machines from backup. Hopefully, this should not happen except in the case of a catastrophic data loss bug in Ganeti or drbd.

Reference

Installation

New node

  • To create a new box, follow new-machine-hetzner-robot but change the following settings:

    • Server: PX62-NVMe
    • Location: FSN1
    • Operating system: Rescue
    • Additional drives: 2x10TB
    • Add in the comment form that the server needs to be in the same datacenter as the other machines (FSN1-DC13, but double-check)
  • follow the new-machine post-install configuration

  • Network setup:

    1. Add the server to the two vSwitch systems in Hetzner Robot web UI

    2. install openvswitch and allow modules to be loaded

      touch /etc/no_modules_disabled
      reboot
      apt install openvswitch-switch
      
    3. Allocate a private IP address in the 30.172.in-addr.arpa zone for the node.

    4. copy over the /etc/network/interfaces from another ganeti node, changing the address and gateway fields to match the local entry.

    5. knock on wood, cross your fingers, pet a cat, help your local book store, and reboot:

      reboot
      
  • Prepare all the nodes by configuring them in puppet. They should be in the class roles::ganeti::fsn if they are part of the fsn cluster.

  • Re-enable modules disabling:

      rm /etc/no_modules_disabled
    
  • reboot again

Then the node is ready to be added to the cluster, by running this on the master node:

puppet agent -t
gnt-node add \
  --secondary-ip 172.30.135.2 \
  --no-ssh-key-check \
  --no-node-setup \
  fsn-node-02.torproject.org

If this is an entirely new cluster, you need a different procedure:

gnt-cluster init \
  --master-netdev vlan-gntbe \
  --vg-name vg_ganeti \
  --secondary-ip 172.30.135.1 \
  --enabled-hypervisors kvm \
  --nic-parameters link=br0,vlan=4000 \
  --mac-prefix 00:66:37 \
  --no-ssh-init \
  --no-etc-hosts \
  fsngnt.torproject.org

The above assumes that fsngnt is already in DNS.

cluster config

These could probably be merged into the cluster init, but just to document what has been done:

gnt-cluster modify --reserved-lvs vg_ganeti/root,vg_ganeti/swap
gnt-cluster modify -H kvm:kernel_path=,initrd_path=,
gnt-cluster modify -H kvm:security_model=pool
gnt-cluster modify -H kvm:kvm_extra='-device virtio-rng-pci\,bus=pci.0\,addr=0x1e\,max-bytes=1024\,period=1000'
gnt-cluster modify -H kvm:disk_cache=none
gnt-cluster modify -H kvm:disk_discard=unmap
gnt-cluster modify -H kvm:scsi_controller_type=virtio-scsi-pci
gnt-cluster modify -H kvm:disk_type=scsi-hd
gnt-cluster modify --uid-pool 4000-4019
gnt-cluster modify --nic-parameters mode=openvswitch,link=br0,vlan=4000
gnt-cluster modify -D drbd:c-plan-ahead=0,disk-custom='--c-plan-ahead 0'
gnt-cluster modify -H kvm:migration_bandwidth=950
gnt-cluster modify -H kvm:migration_downtime=500

Network configuration

IP allocation is managed by Ganeti through the gnt-network(8) system. Say we have 192.0.2.0/24 reserved for the cluster, with the host IP 192.0.2.100`` and the gateway on192.0.2.1`. You will create this network with:

gnt-network add --network 192.0.2.0/24 --gateway 192.0.2.1 --network6 2001:db8::/32 --gateway6 fe80::1 example-network

Then we associate the new network to the default node group:

gnt-network connect --nic-parameters=link=br0,vlan=4000,mode=openvswitch example-network default

The arguments to --nic-parameters come from the values configured in the cluster, above. The current values can be found with gnt-cluster info.

SLA

As long as the cluster is not over capacity, it should be able to survive the loss of a node in the cluster unattended.

Justified machines can be provisionned within a few business days without problems.

New nodes can be provisioned within a week or two, depending on budget and hardware availability.

Design

Our first Ganeti cluster (gnt-fsn) is made of multiple machines hosted with Hetzner Robot, Hetzner's dedicated server hosting service. All machines use the same hardware to avoid problems with live migration. That is currently a customized build of the PX62-NVMe line.

Network layout

Machines are interconnected over a vSwitch, a "virtual layer 2 network" probably implemented using Software-defined Networking (SDN) on top of Hetzner's network. The details of that implementation do not matter much to us, since we do not trust the network and run an IPsec layer on top of the vswitch. We communicate with the vSwitch through Open vSwitch (OVS), which is (currently manually) configured on each node of the cluster.

There are two distinct IPsec networks:

  • gnt-fsn-public: the public network, which maps to the fsn-gnt-inet-vlan vSwitch at Hetzner, the vlan-gntinet OVS network, and the gnt-fsn network pool in Ganeti. it provides public IP addresses and routing across the network. instances get IP allocated in this network.

  • gnt-fsn-be: the private ganeti network which maps to the fsn-gnt-backend-vlan vSwitch at Hetzner and the vlan-gntbe OVS network. it has no matching gnt-network component and IP addresses are allocated manually in the 172.30.135.0/24 network through DNS. it provides internal routing for Ganeti commands and drbd storage mirroring.

Hardware variations

We considered experimenting with the new AX line (AX51-NVMe) but in the past DSA had problems live-migrating (it wouldn't immediately fail but there were "issues" after). So we might need to failover instead of migrate between those parts of the cluster. There are also doubts that the Linux kernel supports those shiny new processors at all: similar processors had trouble booting before Linux 5.5 for example, so it might be worth waiting a little before switching to that new platform, even if it's cheaper. See the cluster configuration section below for a larger discussion of CPU emulation.

CPU emulation

Note that we might want to tweak the cpu_type parameter. By default, it emulates a lot of processing that can be delegated to the host CPU instead. If we use kvm:cpu_type=host, then each node will tailor the emulation system to the CPU on the node. But that might make the live migration more brittle: VMs or processes can crash after a live migrate because of a slightly different configuration (microcode, CPU, kernel and QEMU versions all play a role). So we need to find the lowest common demoninator in CPU families. The list of available families supported by QEMU varies between releases, but is visible with:

# qemu-system-x86_64 -cpu help
Available CPUs:
x86 486
x86 Broadwell             Intel Core Processor (Broadwell)
[...]
x86 Skylake-Client        Intel Core Processor (Skylake)
x86 Skylake-Client-IBRS   Intel Core Processor (Skylake, IBRS)
x86 Skylake-Server        Intel Xeon Processor (Skylake)
x86 Skylake-Server-IBRS   Intel Xeon Processor (Skylake, IBRS)
[...]

The current PX62 line is based on the Coffee Lake Intel micro-architecture. The closest matching family would be Skylake-Server or Skylake-Server-IBRS, according to wikichip. Note that newer QEMU releases (4.2, currently in unstable) have more supported features.

In that context, of course, supporting different CPU manufacturers (say AMD vs Intel) is impractical: they will have totally different families that are not compatible with each other. This will break live migration, which can trigger crashes and problems in the migrated virtual machines.

If there are problems live-migrating between machines, it is still possible to "failover" (gnt-instance failover instead of migrate) which shuts off the machine, fails over disks, and starts it on the other side. That's not such of a big problem: we often need to reboot the guests when we reboot the hosts anyways. But it does complicate our work. Of course, it's also possible that live migrates work fine if no cpu_type at all is specified in the cluster, but that needs to be verified.

Nodes could also grouped to limit (automated) live migration to a subset of nodes.

References:

Installer

The ganeti-instance-debootstrap package is used to install instances. It is configured through Puppet with the shared ganeti module, which deploys a few hooks to automate the install as much as possible. The installer will:

  1. setup grub to respond on the serial console
  2. setup and log a random root password
  3. make sure SSH is installed and log the public keys and fingerprints
  4. setup swap if a labeled partition is present, or a 512MB swapfile otherwise
  5. setup basic static networking through /etc/network/interfaces.d

We have custom configurations on top of that to:

  1. add a few base packages
  2. do our own custom SSH configuration
  3. fix the hostname to be a FQDN
  4. add a line to /etc/hosts
  5. add a tmpfs

There is work underway to refactor and automate the install better, see ticket 31239 for details.

Issues

There is no issue tracker specifically for this project, File or search for issues in the generic internal services component.

Discussion

Overview

The project of creating a Ganeti cluster for Tor has appeared in the summer of 2019. The machines were delivered by Hetzner in July 2019 and setup by weasel by the end of the month.

Goals

The goal was to replace the aging group of KVM servers (kvm[1-5], AKA textile, unifolium, macrum, kvm4 and kvm5).

Must have

  • arbitrary virtual machine provisionning
  • redundant setup
  • automated VM installation
  • replacement of existing infrastructure

Nice to have

  • fully configured in Puppet
  • full high availability with automatic failover
  • extra capacity for new projects

Non-Goals

  • Docker or "container" provisionning - we consider this out of scope for now
  • self-provisionning by end-users: TPA remains in control of provisionning

Approvals required

A budget was proposed by weasel in may 2019 and approved by Vegas in June. An extension to the budget was approved in january 2020 by Vegas.

Proposed Solution

Setup a Ganeti cluster of two machines with a Hetzner vSwitch backend.

Cost

The design based on the PX62 line has the following monthly cost structure:

  • per server: 118EUR (79EUR + 39EUR for 2x10TB HDDs)
  • IPv4 space: 35.29EUR (/27)
  • IPv6 space: 8.40EUR (/64)
  • bandwidth cost: 1EUR/TB (currently 38EUR)

At three servers, that adds up to around 435EUR/mth. Up to date costs are available in the Tor VM hosts.xlsx spreadsheet.

Alternatives considered