Roll call: who's there and emergencies

anarcat and hiro, tons of emergencies:

  • trac spam (#34175)
  • ganeti crash (#34185)

Part-time work schedule

We're splitting the week in two, so far anarcat takes the beginning and hiro the end, for now. This might vary from one week to the next.

The handover, or "change of guard" happens during our weekly mumble meeting, which has been moved to 1400UTC, on Wednesdays.

Roadmap review

We reviewed the sysadmin roadmap at:

Since we're in reduced capacity, the following things were removed from the roadmap:

  • website migration to lektor (#33115) -- should be handled by the "web team"
  • solr search (#33106) -- same, although it does need support from the sysadmin team, we don't have enough cycles for this
  • puppetize nagios (#32901) -- part of the installer automation, not enough time
  • automate installs (#31239) -- same, but moved to october so we can check in progress then

The ganeti cluster work got delayed one month, but we have our spare month to cover for that. We'll let anarcat do the install of fsn-node-06 to get that back on track, but hiro will learn how to setup a new node with (hopefully) fsn-node-07 next.

The listera retirement (#33276), moly migration (#29974) and cymru hardware setup (#29397) are similarly postponed, but hopefully to june (although this will likely carry over to october, if ever).

Next meeting

Change of guard on 1400UTC wednesday May 20th, no minutes.

Formal meetings are switched to the first wednesday of the month, at 1400. So the next formal meeting will be on Wenesday June 3rd at 1400UTC.

Metrics of the month

  • hosts in Puppet: 74, LDAP: 78, Prometheus exporters: 120
  • number of apache servers monitored: 30, hits per second: 164
  • number of nginx servers: 2, hits per second: 2, hit ratio: 0.88
  • number of self-hosted nameservers: 6, mail servers: 10
  • pending upgrades: 0, reboots: 33
  • average load: 0.55, memory available: 358.27 GiB/949.25 GiB, running processes: 383
  • bytes sent: 210.09 MB/s, received: 121.47 MB/s
  • planned buster upgrades completion date: 2020-08-01

Upgrade prediction graph still lives at

Now also available as the main Grafana dashboard. Head to, change the time period to 30 days, and wait a while for results to render.