title: Monitoring kubernetes pod health events
body: <p>Currently I have setup kubernetes, with kube-state-metrics, prometheus and loki. For most things this work really well, but the one thing I am struggling with is finding the exact reason why containers might've restarted.</p>
  <p>For running pods, it's quite easy to see, i.e. with <code>kubectl describe pod</code>, for example, I get the events below:</p>
  <pre><code>kubectl describe pod pod-name
  .....
  Events:
    Type     Reason     Age                  From     Message
    ----     ------     ----                 ----     -------
    Warning  Unhealthy  19m (x9 over 3h29m)  kubelet  message-here
  </code></pre>
  <p>This is very useful for troubleshooting exactly why a container have a lot of restarts, especially if using probes. But there are also other useful events.</p>
  <p>However I cannot see any way to save these kind of events, in either loki or prometheus. But maybe I am missing something. I had expected <code>kube-state-metrics</code> to include such information but it seems not to be the case, I also do not see it anywhere in Loki.</p>
  <p>Any tips on how I can save such events?</p>