Ask HN: How do you monitor your systemd services?
Short answer: Prometheus + Grafana + Alertmanager. prometheus_node_exporter has an option to export SystemD service status and you can alert on failed services, and you can use Alertmanager to configure multiple types of alarms, including repeats so you don&#x27;t forget.<p>Long answer: Whenever I&#x27;ve started to add alerting and monitoring to a system, I end up wanting to add more things each time, so I find it valuable to start from the beginning with an extensible system. For me, Prometheus has been the best option: easy to configure, lightweight, doesn&#x27;t even need to run in the host, and can monitor multiple systems. You just have to configure which exporters you want it to pull data from. In this case, prometheus_node_exporter has a massive amount of stats about a system (including SystemD), and there are default alarms and dashboards out there that will help you create basic monitoring in a minute.<p>You can choose to use Grafana for visualization, and then either the integrated Grafana alerting or use the Prometheus alerting + Prometheus Alertmanager. I think in the latest versions Grafana Alerting includes basically an embedded AlertManager so it should have the same features.<p>Regarding the type of alert itself, I send myself mails for the persistence&#x2F;reminders + Telegram messages for the instant notifications. I find it the best option tbh.