Answers: <blockquote>
  <p>3 Insufficient memory, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't satisfy existing pods anti-affinity rules</p>
  </blockquote>
  <p>This means that ES trying to find a different node to deploy separately all the pods of ES. But the cause of your node count is not enough to run one pod on each node, the other pods remain pending state.</p>
  <p>For more information read <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/" rel="nofollow noreferrer">here</a></p>
  <p>So from here, you have a 2 choice Neo))</p>
  <ol>
  <li>Add more nodes until all pods will have been scheduled on the different nodes</li>
  <li>Minimize your ES sts pod count to 3</li>
  </ol>
  
Answers: CC BY-SA 4.0
Answers: 2021-03-07T18:21:14.030Z
Answers: Is there not a way to have more than one es data pod running on each of the r5 instances? I specifically got the r5.2xl's so that I could run up to roughly 8 es data pods. Other general usage nodes that I am running can run more than one of any kind of pod if there is enough node resources. What makes these tainted nodes only able to run one es data pod at a time. It shouldn't be the nodes resources from what I'm seeing.
Answers: protobyte
Answers: CC BY-SA 4.0
Answers: 2021-03-07T18:46:17.890Z
Answers: You can run more than one pod but it's highly not recommended and doesn't lie under the best practices of DevOps, mainly due to the high availability and fault tolerance.
Answers: Hakob
Answers: CC BY-SA 4.0
Answers: 2021-03-07T19:02:14.147Z
Answers: Ok. That is fair enough. It's mostly trying to test right now to make sure that we can autoscale the pods up before scaling out the nodes. Also that is why there are three different r5 nodes also running es data pods. So one could go down and I'd still have 2 more. So aside from not recommended, and k8s doing this on other nodes/pods by default. Can you explain what it is in my setup I've described which is preventing it from scheduling more than one of the same pod per node?
Answers: protobyte
Answers: CC BY-SA 4.0
Answers: 2021-03-07T19:06:34.487Z
Answers: https://devops.stackexchange.com/a/13497/10769
  
  do you read the link above?
Answers: Hakob
Answers: CC BY-SA 4.0
Answers: 2021-03-07T19:37:47.883Z
Answers: If I'm understanding it correctly. I need to change the `nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution` to `nodeSelector`?
Answers: protobyte
Answers: CC BY-SA 4.0
Answers: 2021-03-07T19:40:41.820Z
Answers: Probably, it is. But again, I personally do not recommend doing it that way
Answers: Hakob
Answers: CC BY-SA 4.0
Answers: 2021-03-08T20:44:44.150Z
Answers: Ok, finally had a chance to make this change. I can confirm that this is what was prohibiting me from scheduling more than one of the same pod to the same node. Thank you for your help and thank you for the best practices recommendation. It is more a clients request. I will stress this point to them once again that it is not recommended. Thank you.
Answers: protobyte
Answers: CC BY-SA 4.0
Answers: 2021-03-08T20:55:07.567Z
Answers: I am excited to hear from you that my advice was to help you. You are welcome.
Answers: Hakob
Answers: CC BY-SA 4.0
Answers: 2021-03-06T19:19:01.113Z
Answers: 2021-03-07T19:09:09.560Z
Answers: 2021-03-07T19:09:09.560Z
Answers: Hakob
Body: <p>I have an elasticsearch cluster in a kubernetes cluster. I have the data pods going to memory optimized nodes which are tainted so that only the elasticsearch data pods get scheduled to the. Right now I have 3 memory optimized ec2 instances for these data pods. They are r5.2Xlarge's which have 64G of memory. Here is the output of one of these r5 nodes. (they all look the same)</p>
  <pre><code>Capacity:
    attachable-volumes-aws-ebs:  25
    cpu:                         8
    ephemeral-storage:           32461564Ki
    hugepages-1Gi:               0
    hugepages-2Mi:               0
    memory:                      65049812Ki
    pods:                        110
  Allocatable:
    attachable-volumes-aws-ebs:  25
    cpu:                         8
    ephemeral-storage:           29916577333
    hugepages-1Gi:               0
    hugepages-2Mi:               0
    memory:                      64947412Ki
    pods:                        110
  System Info:
    Machine ID:                 ec223b5ea23ea6bd5b06e8ed0a733d2d
    System UUID:                ec223b5e-a23e-a6bd-5b06-e8ed0a733d2d
    Boot ID:                    798aca5f-d9e1-4c9f-b75d-e16f7ba2d514
    Kernel Version:             5.4.0-1024-aws
    OS Image:                   Ubuntu 20.04.1 LTS
    Operating System:           linux
    Architecture:               amd64
    Container Runtime Version:  docker://19.3.11
    Kubelet Version:            v1.18.10
    Kube-Proxy Version:         v1.18.10
  Non-terminated Pods:          (5 in total)
    Namespace                   Name                                                        CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
    ---------                   ----                                                        ------------  ----------  ---------------  -------------  ---
    amazon-cloudwatch           fluentd-cloudwatch-tzsv4                                    100m (1%)     0 (0%)      200Mi (0%)       400Mi (0%)     21d
    default                     prometheus-prometheus-node-exporter-tvmd4                   100m (1%)     0 (0%)      0 (0%)           0 (0%)         21d
    es                          elasticsearch-data-0                                        500m (6%)     1 (12%)     8Gi (12%)        8Gi (12%)      14m
    kube-system                 calico-node-dhxg5                                           100m (1%)     0 (0%)      0 (0%)           0 (0%)         21d
    kube-system                 kube-proxy-ip-10-1-12-115.us-gov-west-1.compute.internal    100m (1%)     0 (0%)      0 (0%)           0 (0%)         21d
  Allocated resources:
    (Total limits may be over 100 percent, i.e., overcommitted.)
    Resource                    Requests      Limits
    --------                    --------      ------
    cpu                         900m (11%)    1 (12%)
    memory                      8392Mi (13%)  8592Mi (13%)
    ephemeral-storage           0 (0%)        0 (0%)
    hugepages-1Gi               0 (0%)        0 (0%)
    hugepages-2Mi               0 (0%)        0 (0%)
    attachable-volumes-aws-ebs  0             0
  </code></pre>
  <p>Here is what my cluster looks like</p>
  <pre><code>kubectl get pods -n es
  NAME                                                 READY   STATUS    RESTARTS   AGE
  elasticsearch-client-0                               1/1     Running   0          77m
  elasticsearch-client-1                               1/1     Running   0          77m
  elasticsearch-data-0                                 1/1     Running   0          77m
  elasticsearch-data-1                                 1/1     Running   0          77m
  elasticsearch-data-2                                 1/1     Running   0          77m
  elasticsearch-data-3                                 0/1     Pending   0          77m
  elasticsearch-data-4                                 0/1     Pending   0          77m
  elasticsearch-data-5                                 0/1     Pending   0          77m
  elasticsearch-data-6                                 0/1     Pending   0          77m
  elasticsearch-data-7                                 0/1     Pending   0          77m
  elasticsearch-master-0                               2/2     Running   0          77m
  elasticsearch-master-1                               2/2     Running   0          77m
  prometheus-elasticsearch-exporter-6d6c5d49cf-4w7gc   1/1     Running   0          22h
  </code></pre>
  <p>Here is the events when I describe pod</p>
  <pre><code>Events:
    Type     Reason            Age                  From               Message
    ----     ------            ----                 ----               -------
    Warning  FailedScheduling  56s (x5 over 3m35s)  default-scheduler  0/11 nodes are available: 3 Insufficient memory, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't satisfy existing pods anti-affinity rules, 5 node(s) didn't match node selector.
  </code></pre>
  <p>Here is my resource limits and requests for the data pods</p>
  <pre><code>    Limits:
        cpu:     1
        memory:  8Gi
      Requests:
        cpu:      500m
        memory:   8Gi
  </code></pre>
  <p>Here is what my nodeAffinity looks like</p>
  <pre><code>nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: es-data
          operator: In
          values:
          - &quot;true&quot;
  </code></pre>
  <p>And my tolerations</p>
  <pre><code>tolerations:
    - key: &quot;es-data&quot;
      operator: &quot;Equal&quot;
      value: &quot;true&quot;
      effect: &quot;NoSchedule&quot;
  </code></pre>
  <p>And here is my node tolerance when I describe the node</p>
  <pre><code>Taints:             es-data=true:NoSchedule
  </code></pre>
  <p>I tainted it like</p>
  <pre><code>kubectl taint nodes &lt;node&gt; es-data=true:NoSchedule
  </code></pre>
  <p>According to my calculations based on my understand (which is probably wrong) my data pods are only asking for 8G of memory from a node which has 64G available, and only one pod requesting 8G of memory is already using it. So it should have theoretically 56G of memory left to other pods requesting to be scheduled to it. And even the memory being used shows me it's only 13% used. Why can't it schedule? How can I troubleshoot? Am I misunderstanding how this should work? What else can I tell you which helps troubleshoot this?</p>
  <p>Resolution: Based on Hakob's comments, the issue is that I had <code>nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution</code> set, which is a hard requirement directing the scheduler to only schedule one to each node. What I needed to do in order to schedule more than one to each node was change it to <code>nodeSelector</code>. If you are reaching this, please note Hakob's recommendation for why it is not suggested best practice to do this. I agree with that advice. Though in my case it is a requirement coming from client and did not have the option even after have discussion why they should not be doing this. So please take that into consideration when applying this change.</p>
  
Comments: CC BY-SA 4.0
Comments: 2021-03-06T10:20:20.850Z
Comments: Probably the anti-affinity rule trigger the memory rule along it. I'd first work on the affinity problem first.
Comments: Tensibai
Comments: CC BY-SA 4.0
Comments: 2021-03-06T16:56:43.613Z
Comments: Could you elaborate please. The affinity should be ok, since the first three are getting scheduled to the appropriate memory optimized tainted nodes just fine. Shouldn't they? And they never get scheduled where they are not suppose to. No other pods are scheduled to these memory optimized nodes. So everything seems just as I expected, until more than 1 of these pods try to get scheduled to one of the memory optimized.
Comments: protobyte
Comments: CC BY-SA 4.0
Comments: 2021-03-07T23:35:13.040Z
Comments: Roughly what is in the answer: you ask for your pods to NOT be on the same node, so obviously it can only start as much pods as you have nodes.
Comments: Tensibai
Tags: <kubernetes>
Title: Why is there insufficient memory on kubernetes node
UserDisplayName: protobyte
ThreadText: # Why is there insufficient memory on kubernetes node
  
  (Asked by: username_0 on 2021-03-04)
  
  I have an elasticsearch cluster in a kubernetes cluster. I have the data pods going to memory optimized nodes which are tainted so that only the elasticsearch data pods get scheduled to the. Right now I have 3 memory optimized ec2 instances for these data pods. They are r5.2Xlarge's which have 64G of memory. Here is the output of one of these r5 nodes. (they all look the same)
  
  ```
  Capacity:
    attachable-volumes-aws-ebs:  25
    cpu:                         8
    ephemeral-storage:           32461564Ki
    hugepages-1Gi:               0
    hugepages-2Mi:               0
    memory:                      65049812Ki
    pods:                        110
  Allocatable:
    attachable-volumes-aws-ebs:  25
    cpu:                         8
    ephemeral-storage:           29916577333
    hugepages-1Gi:               0
    hugepages-2Mi:               0
    memory:                      64947412Ki
    pods:                        110
  System Info:
    Machine ID:                 ec223b5ea23ea6bd5b06e8ed0a733d2d
    System UUID:                ec223b5e-a23e-a6bd-5b06-e8ed0a733d2d
    Boot ID:                    798aca5f-d9e1-4c9f-b75d-e16f7ba2d514
    Kernel Version:             5.4.0-1024-aws
    OS Image:                   Ubuntu 20.04.1 LTS
    Operating System:           linux
    Architecture:               amd64
    Container Runtime Version:  docker://19.3.11
    Kubelet Version:            v1.18.10
    Kube-Proxy Version:         v1.18.10
  Non-terminated Pods:          (5 in total)
    Namespace                   Name                                                        CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
    ---------                   ----                                                        ------------  ----------  ---------------  -------------  ---
    amazon-cloudwatch           fluentd-cloudwatch-tzsv4                                    100m (1%)     0 (0%)      200Mi (0%)       400Mi (0%)     21d
    default                     prometheus-prometheus-node-exporter-tvmd4                   100m (1%)     0 (0%)      0 (0%)           0 (0%)         21d
    es                          elasticsearch-data-0                                        500m (6%)     1 (12%)     8Gi (12%)        8Gi (12%)      14m
    kube-system                 calico-node-dhxg5                                           100m (1%)     0 (0%)      0 (0%)           0 (0%)         21d
    kube-system                 kube-proxy-ip-10-1-12-115.us-gov-west-1.compute.internal    100m (1%)     0 (0%)      0 (0%)           0 (0%)         21d
  Allocated resources:
    (Total limits may be over 100 percent, i.e., overcommitted.)
    Resource                    Requests      Limits
    --------                    --------      ------
    cpu                         900m (11%)    1 (12%)
    memory                      8392Mi (13%)  8592Mi (13%)
    ephemeral-storage           0 (0%)        0 (0%)
    hugepages-1Gi               0 (0%)        0 (0%)
    hugepages-2Mi               0 (0%)        0 (0%)
    attachable-volumes-aws-ebs  0             0
  
  ```
  
  Here is what my cluster looks like
  
  ```
  kubectl get pods -n es
  NAME                                                 READY   STATUS    RESTARTS   AGE
  elasticsearch-client-0                               1/1     Running   0          77m
  elasticsearch-client-1                               1/1     Running   0          77m
  elasticsearch-data-0                                 1/1     Running   0          77m
  elasticsearch-data-1                                 1/1     Running   0          77m
  elasticsearch-data-2                                 1/1     Running   0          77m
  elasticsearch-data-3                                 0/1     Pending   0          77m
  elasticsearch-data-4                                 0/1     Pending   0          77m
  elasticsearch-data-5                                 0/1     Pending   0          77m
  elasticsearch-data-6                                 0/1     Pending   0          77m
  elasticsearch-data-7                                 0/1     Pending   0          77m
  elasticsearch-master-0                               2/2     Running   0          77m
  elasticsearch-master-1                               2/2     Running   0          77m
  prometheus-elasticsearch-exporter-6d6c5d49cf-4w7gc   1/1     Running   0          22h
  
  ```
  
  Here is the events when I describe pod
  
  ```
  Events:
    Type     Reason            Age                  From               Message
    ----     ------            ----                 ----               -------
    Warning  FailedScheduling  56s (x5 over 3m35s)  default-scheduler  0/11 nodes are available: 3 Insufficient memory, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't satisfy existing pods anti-affinity rules, 5 node(s) didn't match node selector.
  
  ```
  
  Here is my resource limits and requests for the data pods
  
  ```
      Limits:
        cpu:     1
        memory:  8Gi
      Requests:
        cpu:      500m
        memory:   8Gi
  
  ```
  
  Here is what my nodeAffinity looks like
  
  ```
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: es-data
          operator: In
          values:
          - "true"
  
  ```
  
  And my tolerations
  
  ```
  tolerations:
    - key: "es-data"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
  
  ```
  
  And here is my node tolerance when I describe the node
  
  ```
  Taints:             es-data=true:NoSchedule
  
  ```
  
  I tainted it like
  
  ```
  kubectl taint nodes <node> es-data=true:NoSchedule
  
  ```
  
  According to my calculations based on my understand (which is probably wrong) my data pods are only asking for 8G of memory from a node which has 64G available, and only one pod requesting 8G of memory is already using it. So it should have theoretically 56G of memory left to other pods requesting to be scheduled to it. And even the memory being used shows me it's only 13% used. Why can't it schedule? How can I troubleshoot? Am I misunderstanding how this should work? What else can I tell you which helps troubleshoot this?
  
  Resolution: Based on username_2's comments, the issue is that I had `nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution` set, which is a hard requirement directing the scheduler to only schedule one to each node. What I needed to do in order to schedule more than one to each node was change it to `nodeSelector`. If you are reaching this, please note username_2's recommendation for why it is not suggested best practice to do this. I agree with that advice. Though in my case it is a requirement coming from client and did not have the option even after have discussion why they should not be doing this. So please take that into consideration when applying this change.
  
  **Tags:** kubernetes
  
  > **Comments:**
  > **username_1**: Probably the anti-affinity rule trigger the memory rule along it. I'd first work on the affinity problem first.  
  > **username_0**: Could you elaborate please. The affinity should be ok, since the first three are getting scheduled to the appropriate memory optimized tainted nodes just fine. Shouldn't they? And they never get scheduled where they are not suppose to. No other pods are scheduled to these memory optimized nodes. So everything seems just as I expected, until more than 1 of these pods try to get scheduled to one of the memory optimized.  
  > **username_1**: Roughly what is in the answer: you ask for your pods to NOT be on the same node, so obviously it can only start as much pods as you have nodes.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-03-06. Score: 2 | Accepted answer
  
  > 3 Insufficient memory, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't satisfy existing pods anti-affinity rules
  
  This means that ES trying to find a different node to deploy separately all the pods of ES. But the cause of your node count is not enough to run one pod on each node, the other pods remain pending state.
  
  For more information read [here](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/)
  
  So from here, you have a 2 choice Neo))
  
  1. Add more nodes until all pods will have been scheduled on the different nodes
  2. Minimize your ES sts pod count to 3
  
  > **Comments:**
  > **username_0**: Is there not a way to have more than one es data pod running on each of the r5 instances? I specifically got the r5.2xl's so that I could run up to roughly 8 es data pods. Other general usage nodes that I am running can run more than one of any kind of pod if there is enough node resources. What makes these tainted nodes only able to run one es data pod at a time. It shouldn't be the nodes resources from what I'm seeing.  
  > **username_2**: You can run more than one pod but it's highly not recommended and doesn't lie under the best practices of DevOps, mainly due to the high availability and fault tolerance.  
  > **username_0**: Ok. That is fair enough. It's mostly trying to test right now to make sure that we can autoscale the pods up before scaling out the nodes. Also that is why there are three different r5 nodes also running es data pods. So one could go down and I'd still have 2 more. So aside from not recommended, and k8s doing this on other nodes/pods by default. Can you explain what it is in my setup I've described which is preventing it from scheduling more than one of the same pod per node?  
  > **username_2**: https://devops.stackexchange.com/a/13497/10769  do you read the link above?  
  > **username_0**: If I'm understanding it correctly. I need to change the `nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution` to `nodeSelector`?  
  > **username_2**: Probably, it is. But again, I personally do not recommend doing it that way  
  > **username_0**: Ok, finally had a chance to make this change. I can confirm that this is what was prohibiting me from scheduling more than one of the same pod to the same node. Thank you for your help and thank you for the best practices recommendation. It is more a clients request. I will stress this point to them once again that it is not recommended. Thank you.  
  > **username_2**: I am excited to hear from you that my advice was to help you. You are welcome.  
  
  ---
  
  