title: Liveness Probe, Readiness Probe not called in expected duration
body: <p>On GKE, I tried to use readiness probe/ liveness probe , and post alert using monitoring <a href="https://cloud.google.com/monitoring/alerts/using-alerting-ui" rel="nofollow noreferrer">https://cloud.google.com/monitoring/alerts/using-alerting-ui</a></p>
  <p>as a test, I create a pod which has readiness probe/ liveness probe. Probe check failed everytime, as I expected.</p>
  <pre><code>apiVersion: v1
  kind: Pod
  metadata:
    labels:
      test: liveness
    name: liveness-http
  spec:
    containers:
    - name: liveness
      image: k8s.gcr.io/liveness
      args:
      - /server
      readinessProbe:
        httpGet:
          path: /healthz
          port: 8080
          httpHeaders:
          - name: X-Custom-Header
            value: Awesome
        initialDelaySeconds: 0
        periodSeconds: 10      
        timeoutSeconds: 10
        successThreshold: 1
        failureThreshold: 3
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
          httpHeaders:
          - name: X-Custom-Header
            value: Awesome
        initialDelaySeconds: 20
        periodSeconds: 60
        timeoutSeconds: 30      
        successThreshold: 1
        failureThreshold: 3 
  </code></pre>
  <p>And checking GCP log, both error logs showed up based on periodSeconds first.</p>
  <p>Readiness probe: every 10 seconds</p>
  <blockquote>
  <p>2021-02-21 13:26:30.000 JST
  Readiness probe failed: HTTP probe failed with statuscode: 500</p>
  </blockquote>
  <blockquote>
  <p>2021-02-21 13:26:40.000 JST
  Readiness probe failed: HTTP probe failed with statuscode: 500</p>
  </blockquote>
  <p>Liveness probe: every 1 minute</p>
  <blockquote>
  <p>2021-02-21 13:25:40.000 JST
  Liveness probe failed: HTTP probe failed with statuscode: 500</p>
  </blockquote>
  <blockquote>
  <p>2021-02-21 13:26:40.000 JST
  Liveness probe failed: HTTP probe failed with statuscode: 500</p>
  </blockquote>
  <p>But, after running this pod several minutes</p>
  <ul>
  <li>Liveness probe check didn't not called anymore</li>
  <li>Readiness probe check called but interval became long ( maximum interval looks about 10 minutes)</li>
  </ul>
  <pre><code>$ kubectl get event
  LAST SEEN   TYPE      REASON      OBJECT              MESSAGE
  30m         Normal    Pulling     pod/liveness-http   Pulling image &quot;k8s.gcr.io/liveness&quot;
  25m         Warning   Unhealthy   pod/liveness-http   Readiness probe failed: HTTP probe failed with statuscode: 500
  20m         Warning   BackOff     pod/liveness-http   Back-off restarting failed container
  20m         Normal    Scheduled   pod/liveness-http   Successfully assigned default/liveness-http to gke-cluster-default-pool-8bc9c75c-rfgc
  17m         Normal    Pulling     pod/liveness-http   Pulling image &quot;k8s.gcr.io/liveness&quot;
  17m         Normal    Pulled      pod/liveness-http   Successfully pulled image &quot;k8s.gcr.io/liveness&quot;
  17m         Normal    Created     pod/liveness-http   Created container liveness
  20m         Normal    Started     pod/liveness-http   Started container liveness
  4m59s       Warning   Unhealthy   pod/liveness-http   Readiness probe failed: HTTP probe failed with statuscode: 500
  17m         Warning   Unhealthy   pod/liveness-http   Liveness probe failed: HTTP probe failed with statuscode: 500
  17m         Normal    Killing     pod/liveness-http   Container liveness failed liveness probe, will be restarted
  </code></pre>
  <hr />
  <p>In my plan, I would create alert policy, whose condition is like</p>
  <ul>
  <li>if liveness probe error happens 3 times in 3 minutes</li>
  </ul>
  <p>but if probe check didn't called as I expect, these policy didn't work; even if pod is not running, alert became fixed</p>
  <hr />
  <p>Why Liveness probe didn't run, and interval of Readiness probe changed ?</p>
  <p>Note: if there are other good alert policy to check liveness of pod, I would not care that behavior. I appreciate it if someone would advice me what kind of alert policy is ideal to check pod.</p>