title: How to set up an alert when Liveness/Readiness probe fails in Kubernetes
body: <p>I am trying to set up alerts/monitors for my Kubernetes PODs where my application is deployed.</p>
  <p>I am using <strong>Kubernetes=1.15</strong>.</p>
  <p>I am trying to set up alert for following two scenarios:</p>
  <ol>
  <li>When any POD goes down due to some reason, alert should get triggered.</li>
  <li>When my application goes down on any of the POD, alert should get triggered.</li>
  </ol>
  <p>For (1) I am setting up an alert if Liveness probe failed &gt;= 1, if all you care about is one or more pods going down</p>
  <p>For (2), I am setting up an alert on Readiness probe_type and result as failed &gt;= 1</p>
  <p>For (1), I am using <strong>kubelet:prober_probe_total{probe_type=Readiness,result=failed,..}</strong></p>
  <p>For (2), I am using <strong>kubelet:prober_probe_total{probe_type=Liveness,result=failed,..}</strong></p>
  <p>Reference:</p>
  <ol>
  <li><p><a href="https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/prober/prober_manager.go" rel="nofollow noreferrer">https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/prober/prober_manager.go</a></p>
  </li>
  <li><p><a href="https://godoc.org/k8s.io/kubernetes/pkg/kubelet/prober" rel="nofollow noreferrer">https://godoc.org/k8s.io/kubernetes/pkg/kubelet/prober</a></p>
  </li>
  </ol>
  <p>I have been thinking that an alert should get triggered when I get any non zero value for them. But even when everything is fine, I am getting non-zero data.</p>
  <p>So I am clueless here.</p>