Procedure

  1. Preperation:

    : reset to the default locale
    export LC_ALL=C.UTF-8 &&
    sudo apt install ttyrec screen debconf-utils apt-show-versions deborphan &&
    sudo ttyrec -e screen /var/log/upgrade-buster.ttyrec
    
  2. Backups and checks:

    ( umask 0077 &&
      tar cfz /var/backups/pre-buster-backup.tgz /etc /var/lib/dpkg /var/lib/apt/extended_states /var/lib/aptitude/pkgstates /var/cache/debconf &&
      dpkg --get-selections "*" > /var/backups/dpkg-selections-pre-buster.txt &&
      debconf-get-selections > /var/backups/debconf-selections-pre-buster.txt
    ) &&
    apt-mark showhold &&
    dpkg --audit &&
    : look for dkms packages and make sure they are relevant, if not, purge. &&
    ( dpkg -l '*dkms' || true ) &&
    : make sure backups are up to date in Nagios &&
    echo End of Step 2
    
  3. Perform any pending upgrade and clear out old pins:

    : Check for pinned, on hold, packages, and possibly disable &&
    rm -f /etc/apt/preferences /etc/apt/preferences.d/* &&
    rm -f /etc/apt/sources.list.d/testing.list &&
    rm -f /etc/apt/sources.list.d/stretch-backports.list &&
    rm -f /etc/apt/sources.list.d/backports.debian.org.list &&
    apt update && apt -y upgrade &&
    : list kernel images and purge unused packages &&
    dpkg -l 'linux-image-*' &&
    : look for packages from backports, other suites or archives &&
    : if possible, switch to official packages by disabling third-party repositories &&
    apt-show-versions | grep -v /stretch | grep -v 'not installed$' &&
    echo End of Step 3
    
  4. Check free space, see this guide to free up space and download packages:

    sed -i 's/stretch/buster/g' /etc/apt/sources.list.d/* &&
    apt update && apt -o APT::Get::Trivial-Only=true dist-upgrade &&
    df -h &&
    apt -y -d upgrade &&
    apt -y -d dist-upgrade &&
    echo End of Step 4
    
  5. Enable module loading (for ferm)

    sed -i -e 's/.*modules_disabled/#&/' /etc/rc.local
    
    reboot
    
    export LC_ALL=C.UTF-8 &&
    sudo ttyrec -e screen /var/log/upgrade-buster.ttyrec.2
    
  6. Actual upgrade run:

    apt install -y dpkg apt &&
    apt install ferm &&
    apt dist-upgrade &&
    echo 'End of Step 6'
    
  7. Post-upgrade procedures:

    apt-get update --allow-releaseinfo-change &&
    apt-mark manual git &&
    apt --purge autoremove &&
    apt purge $(for i in apt-transport-https dh-python emacs24-nox gnupg-agent libbind9-140 libcryptsetup4 libdns-export162 libdns162 libevent-2.0-5 libevtlog0 libgdbm3 libicu57 libisc-export160 libisc160 libisccc140 libisccfg140 liblvm2app2.2 liblvm2cmd2.02 liblwres141 libmpfr4 libncurses5 libperl5.24 libprocps6 libpython3.5 libpython3.5-minimal libpython3.5-stdlib libruby2.3 libssl1.0.2 libunbound2 libunistring0 python3-distutils python3-lib2to3 python3.5 python3.5-minimal ruby-nokogiri ruby-pkg-config ruby-rgen ruby-safe-yaml ruby2.3 sgml-base xml-core git-core gcc-6-base:amd64 nagios-plugins-basic perl-modules-5.24 libsensors4:amd64 grub2 iproute libncursesw5 libustr-1.0-1; do dpkg -l "$i" 2>/dev/null | grep -q '^ii' && echo "$i"; done) &&
    dpkg --purge libsensors4:amd64 syslog-ng-mod-json
    
    (puppet agent -t || true) &&
    (puppet agent -t || true)
    
  8. reboot once so we can then get rid of the old kernel image

    reboot
    
  9. Post-upgrade checks:

    export LC_ALL=C.UTF-8 &&
    sudo ttyrec -e screen /var/log/upgrade-buster.ttyrec.3
    
    # review and purge old packages, including kernels
    apt --purge autoremove
    dsa-check-packages | tr -d ,
    while deborphan -n | grep -q . ; do apt purge $(deborphan -n); done
    apt --purge autoremove
    dpkg -l '*-dbg' # look for dbg package and possibly replace with -dbgsym
    apt clean
    # review packages that are not in the new distribution
    apt-show-versions | grep -v /buster
    reboot
    
  10. Change the hostgroup of the host to buster in Nagios (in tor-nagios/config/nagios-master.cfg on git@git-rw.tpo)

Issues

Pending

  • upgrading restarts openvswitch will mean all guests lose network

  • At least on kvm5, brpub was having issues. Either ipv4 or ipv6 address was missing, or the v6 route to the guests was missing. Probably because the ipv6 route setting failed since we set a prefsrc and that was only brought up later?

    Rewrote /etc/network/interfaces to set things up more manually. On your host, check if brpub has both ipv4 and ipv6 addresses after boot before launching VMs, and that is has an ipv6 route into brpub with the configured prefsrc address. If not, fiddle likewise.

    See ticket #31083 for followup on possible routing issues.

  • On physical hosts witch /etc/sysfs.d/local-io-schedulers.conf, note that deadline no longer existsts. Probably it is also not necessary as Linux might pick the right scheduler anyhow.

  • the following config files had conflicts but were managed by Puppet so those changes were ignored for now. eventually they should be upgraded in Puppet as well.

     /etc/bacula/bacula-fd.conf
     /etc/bind/named.conf.options
     /etc/default/stunnel4
     /etc/ferm/ferm.conf
     /etc/init.d/stunnel4
     /etc/logrotate.d/ulogd2
     /etc/nagios/nrpe.cfg
     /etc/ntp.conf
     /etc/ssh/sshd_config
     /etc/syslog-ng/syslog-ng.conf
    
  • ferm failed to reload during upgrade, with the following error:

     Couldn't load match `state':No such file or directory
    
  • Puppet will warn with the following when running against a stretch master, harmlessly:

     Warning: Downgrading to PSON for future requests
    
  • Puppet might try to downgrade the sources.list files to stretch or n/a for some reason, just re-run Puppet after fixing the sources.list files, it will eventually figure it out.

  • The official list of known issues

Resolved

  • apt-get complains like this after upgrade (bug #929248):

     E: Repository 'https://mirrors.wikimedia.org/debian buster InRelease' changed its 'Suite' value from 'testing' to 'stable'
    

    the following workaround was added to the upgrade instructions, above, but might be necessary on machines where this procedure was followed before the note was added:

     apt-get update --allow-releaseinfo-change
    

References

Note: the official upgrade guide and release notes not available at the time of writing (2019-04-08) as the documentation is usually written during the freeze and buster is not there yet.