Grafana is a graphing engine and dashboard management tool that processes data from multiple data sources. We use it to trend various metrics collected from servers by Prometheus.

Puppet implementation

Grafana was installed with Puppet using the upstream Debian package, following a debate regarding the merits of Debian packages versus Docker containers when neither are trusted, see this comment for a summary.

Some manual configuration was performed after the install. An admin password reset on first install, stored in tor-passwords.git, in hosts-extra-info. Everything else is configured in Puppet.

Grafana dashboards, in particular, live in modules/profile/files/grafana/dashboards, which is a copy of anarcat's grafana-dashboards repository. The file there contains more instructions on how to add and update dashboards. In general, dashboards must not be modified directly through the web interface, at least not without being exported back into the repository.

Updating a Grafana dashboard

As mentioned above, the Grafana dashboards are maintained by Puppet. So while new dashboard can be created and edited in the Grafana web interface, changes will be lost when Puppet ships a new version of the dashboard.

You therefore need to make sure you update the Dashboard in git before leaving. New dashboards not in git should be fairly safe, but please do also commit them to git so we have a proper revisioned history of their deployment. It's also the right way to make sure they are usable across other instances of Grafana. Finally, they are also easier to share and collaborate on that way.

See the grafana-dashboards repository for instructions on how to export dashboards into git.