Answers: <p>To answer your two questions:</p>
  <blockquote>
  <p>I'm Interested before terminating a pod does kuberentes makes sure that that no traffic is load balanced on to be terminated pod.</p>
  </blockquote>
  <p>If you have more than 1 pod running then yes, it will not send traffic to the failed/terminating pod.</p>
  <blockquote>
  <p>Also what are the best practices to fine tune Kubernetes pod eviction.</p>
  </blockquote>
  <p>The <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/" rel="nofollow noreferrer">official Kubernetes documentation</a> explains the Pod lifetime cycle in great detail along with suggestions on how to fine tune.  While <a href="http://liveness%20probes" rel="nofollow noreferrer">liveness probes</a> and <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#when-should-you-use-a-readiness-probe" rel="nofollow noreferrer">readiness probes</a> can help determine the health of a pod, they will not help in the case of eviction.</p>
  <p>Eviction occurs due to a lack of resources (your case), or if the Node is unavailable (e.g., spot instance eviction, Node maintenance).  By using something like a ReplicaSet or Deployment you can specify how many pods you want to have running and can fine tune where these run.  For instance a DaemonSet will by default deploy to every node.  Architecting this way will ensure that if a node goes down, or a pod fails to run due to lack of resources, it will have other nodes available or will be rescheduled to launch on another node.</p>
  
Answers: CC BY-SA 4.0
Answers: 2021-10-05T19:55:43.970Z
Answers: 2021-10-05T19:55:43.970Z
Answers: Wesley Rolnick
Answers: <p>The pod eviction can happen because of resources outages like CPU, ram, and storage.
  You can prevent your cluster from the eviction of pod by declaring the <code>resource quotas</code>.
  You can define resource quota over namespace but before that, you have to calculate workload usage and create a resource limit on provided request quota.
  You have three types of quotas available.</p>
  <ol>
  <li>Compute Quotas</li>
  <li>Storage Quotas</li>
  <li>Object Count Quotas</li>
  </ol>
  <p>There are a few more ways to quotas on the kubernetes cluster.</p>
  <p><a href="https://kubernetes.io/docs/concepts/policy/resource-quotas/" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/policy/resource-quotas/</a></p>
  
Answers: CC BY-SA 4.0
Answers: 2021-09-21T07:55:06.353Z
Answers: 2021-09-21T07:55:06.353Z
Answers: Arvind Rawat
Answers: <p>setup a readiness Probe in the pod spec, something like this</p>
  <pre><code>readinessProbe:
    httpGet:
      path: /
      port: 443
  </code></pre>
  <p>readiness probes specifically inform service routing decisions</p>
  
Answers: CC BY-SA 4.0
Answers: 2021-09-18T17:14:14.277Z
Answers: Readiness probe is already in place.
Answers: Terminator
Answers: CC BY-SA 4.0
Answers: 2021-09-18T15:43:33.967Z
Answers: 2021-09-18T15:43:33.967Z
Answers: Vorsprung
Body: <p>I'm trying understand to Kuberentes pod eviction behaviour and when it decides to terminate a pod how it manages that no request should be in process state when it is going to terminate a pod.</p>
  <p>It is based on a recent troubleshooting of incident where 3 requests resulted in 502 bad gateway on the ingress controller Kong.</p>
  <p>During the analysis it was found out that at the exact same time K8s terminated a pod because it was hitting ephemeral storage limits set on one of the container of the pod. In addition to this on linkerd container logged that connection was reset by peer and proxy was failed to process the request. This connection reset translated to 502 bad gateway for ingress controller kong.</p>
  <p>I understand by improving the limits and controlling the number bytes container writes on disk I can avoid eviction by kubernetes to a great extent.</p>
  <p>I'm Interested before terminating a pod does kuberentes makes sure that that no traffic is load balanced on to be terminated pod.
  Also what are the best practices to fine tune Kubernetes pod eviction.</p>
  
Comments: CC BY-SA 4.0
Comments: 2021-10-08T13:22:38.570Z
Comments: you might like to read this https://kubernetes.io/docs/concepts/scheduling-eviction/_print/
Comments: Vorsprung
Comments: CC BY-SA 4.0
Comments: 2023-06-12T00:36:13.900Z
Comments: @terminator I am facing something similar now, Could you please add your findings or solutions here
Comments: Maram Sreerama Reddy
Tags: <kubernetes>
Title: Kubernetes pod eviction and traffic draining for to be evicted pod
UserDisplayName: Terminator
ThreadText: # Kubernetes pod eviction and traffic draining for to be evicted pod
  
  (Asked by: username_0 on 2021-09-17)
  
  I'm trying understand to Kuberentes pod eviction behaviour and when it decides to terminate a pod how it manages that no request should be in process state when it is going to terminate a pod.
  
  It is based on a recent troubleshooting of incident where 3 requests resulted in 502 bad gateway on the ingress controller Kong.
  
  During the analysis it was found out that at the exact same time K8s terminated a pod because it was hitting ephemeral storage limits set on one of the container of the pod. In addition to this on linkerd container logged that connection was reset by peer and proxy was failed to process the request. This connection reset translated to 502 bad gateway for ingress controller kong.
  
  I understand by improving the limits and controlling the number bytes container writes on disk I can avoid eviction by kubernetes to a great extent.
  
  I'm Interested before terminating a pod does kuberentes makes sure that that no traffic is load balanced on to be terminated pod.
  Also what are the best practices to fine tune Kubernetes pod eviction.
  
  **Tags:** kubernetes
  
  > **Comments:**
  > **username_1**: you might like to read this https://kubernetes.io/docs/concepts/scheduling-eviction/_print/  
  > **username_2**: @username_0 I am facing something similar now, Could you please add your findings or solutions here  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2021-10-05. Score: 1
  
  To answer your two questions:
  
  > I'm Interested before terminating a pod does kuberentes makes sure that that no traffic is load balanced on to be terminated pod.
  
  If you have more than 1 pod running then yes, it will not send traffic to the failed/terminating pod.
  
  > Also what are the best practices to fine tune Kubernetes pod eviction.
  
  The [official Kubernetes documentation](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/) explains the Pod lifetime cycle in great detail along with suggestions on how to fine tune. While [liveness probes](http://liveness%20probes) and [readiness probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#when-should-you-use-a-readiness-probe) can help determine the health of a pod, they will not help in the case of eviction.
  
  Eviction occurs due to a lack of resources (your case), or if the Node is unavailable (e.g., spot instance eviction, Node maintenance). By using something like a ReplicaSet or Deployment you can specify how many pods you want to have running and can fine tune where these run. For instance a DaemonSet will by default deploy to every node. Architecting this way will ensure that if a node goes down, or a pod fails to run due to lack of resources, it will have other nodes available or will be rescheduled to launch on another node.
  
  
  ---
  
  ### Answer by username_4 on 2021-09-21. Score: 1
  
  The pod eviction can happen because of resources outages like CPU, ram, and storage.
  You can prevent your cluster from the eviction of pod by declaring the `resource quotas`.
  You can define resource quota over namespace but before that, you have to calculate workload usage and create a resource limit on provided request quota.
  You have three types of quotas available.
  
  1. Compute Quotas
  2. Storage Quotas
  3. Object Count Quotas
  
  There are a few more ways to quotas on the kubernetes cluster.
  
  <https://kubernetes.io/docs/concepts/policy/resource-quotas/>
  
  
  ---
  
  ### Answer by username_1 on 2021-09-18. Score: 1
  
  setup a readiness Probe in the pod spec, something like this
  
  ```
  readinessProbe:
    httpGet:
      path: /
      port: 443
  
  ```
  
  readiness probes specifically inform service routing decisions
  
  > **Comments:**
  > **username_0**: Readiness probe is already in place.  
  
  ---
  
  