Stack Exchange
Kubernetes node.kubernetes.io/memory-pressure:NoSchedule
Good afternoon
I have two pods that are not lifting due to the following errors:
prometheus-k8s-0 0/3 Pending 0 56s
prometheus-k8s-1 2/3 Running 0 69s

Warning FailedScheduling <unknown> default-scheduler 0/4 nodes are available: 1 node (s) had taint {node.kubernetes.io/memory-pressure:}, that the pod didn't tolerate, 3 node (s) had volume node affinity conflict.
kubectl describe nodes aks-bbagentpool-39730754-vmss000000 | grep -i taint
Taints: node.kubernetes.io/memory-pressure:NoSchedule

I do not see high consumption in any of the nodes:
kubectl top nodes

NAME CPU (cores) CPU% MEMORY (bytes) MEMORY%

aks-bbagentpool-39730754-vmss000000 153m 8% 920Mi 17%
aks-bbagentpool-39730754-vmss000001 1854m 97% 5357Mi 99%
aks-bbinfra-39730754-vmss000000 211m 5% 2038Mi 16%
aks-bbinfra-39730754-vmss000001 111m 2% 1706Mi 13%

The prometheus solution is from Coreos and I see that the daemonset is operational
kubectl get daemonsets.apps -n monitoring

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE

node-exporter 4 4 4 4 4 kubernetes.io/os=linux 3d13h

I want to know the reason for this failure and how I can resolve the status of the nodes.

Taint node.kubernetes.io/memory-pressure will be automatically added by the node controller when there is a no enough memory to schedule any new pods.
In your case the particular node would have been reached memory issue at some point and this taint has been added by node controller to prevent the node from attracting any new pod scheduling, and in due course the memory would have been released and came back to normal. Taints can be applied automatically by controller based on meeting certain condition but it will not be removed automatically.
You got two options to fix this.

remove the taint from the node kubectl taint nodes aks-bbagentpool-39730754-vmss000000 node.kubernetes.io/memory-pressure:NoSchedule-
Or add toleration to your pod (which I personally don't prefer)

You can get more details on Docs and Blog

Hi, I have a question. Strictly speaking, the taint is added from nodelifecycle controller I think. is this correct? ref: https://github.com/kubernetes/kubernetes/blob/release-1.19/pkg/controller/nodelifecycle/node_lifecycle_controller.go#L644-L666

To remove taint from the node use
kubectl taint node <nodename> node.kubernetes.io/memory-pressure:NoSchedule-

Don't forget - at the end
OR another way is like below
If you want to keep the taint on node as it is and still you want to schedule pod on that node then include this in your pod/deployment.yaml file.
  spec:
    containers:
    - name: nginx
      image: nginx
      imagePullPolicy: IfNotPresent
    tolerations: # <---------------------------- This part
    - key: "node.kubernetes.io/memory-pressure"
      operator: "Exists"
      effect: "NoSchedule"


Apply kubectl taint node  node.kubernetes.io/memory-pressure:NoSchedule- remove the tag and put it back, check the status of the nodes and there is no consumption, restart the nodes, restart the kubelet and still the failure. I know I can avoid this in the deploy manifest by setting tolerant but I need to know what happens with the assigned pool since both nodes of the pool mark them with memory-pressure randomly.
