Answers: <p>You will probably find the error in either</p>
  <pre><code>sudo journalctl -u kubelet -f
  </code></pre>
  <p>or</p>
  <pre><code>sudo journalctl -u containerd -f
  </code></pre>
  <p>My guess would be that you have SystemdCgroup set to false in /etc/containerd/config.toml</p>
  <p>Set it to true on all nodes and restart containerd</p>
  <pre><code>SystemdCgroup = true
  </code></pre>
  <p>when you dont have the file yet, create it</p>
  <p><code>containerd config default &gt; /etc/containerd/config.toml</code></p>
  <p>then set SystemdCgroup to true and restart containerd</p>
  
Answers: CC BY-SA 4.0
Answers: 2023-11-03T09:38:37.027Z
Answers: thank you,
  
  I checked the journalctl and found that there is some error related to 'configmap cache sync'. I've added the output of both the command in the question, do you have any idea about the same?
  Also regarding, config.toml file, there is no such file in my system,
Answers: Sharad Tomer
Answers: CC BY-SA 4.0
Answers: 2023-11-03T11:40:48.887Z
Answers: edited the answer to create congig.toml. you are using calico for CNI, try use flannel if the containerd config solution does not work. flannel is easier to begin with but it lacks some features
Answers: Serve Laurijssen
Answers: CC BY-SA 4.0
Answers: 2023-11-03T06:55:31.407Z
Answers: 2023-11-03T11:38:16.693Z
Answers: 2023-11-03T11:38:16.693Z
Answers: Serve Laurijssen
Body: <p>I have a local Kubernetes cluster setup using kubeadm. Everything was working fine a few days back but now I've got a strange problem.</p>
  <p>Whenever I add new resources be it a deployment or pod, it never gets scheduled and is forever stuck in Pending state without any error. Please let me know if you anyone has any idea how to fix this issue or even debug for that matter.</p>
  <p>I'm pasting output of some of the commands I think might be useful, let me know if you need any additional information</p>
  <p>•  kubectl get cs</p>
  <pre><code>NAME                 STATUS    MESSAGE                         ERROR
  scheduler            Healthy   ok
  controller-manager   Healthy   ok
  etcd-0               Healthy   {&quot;health&quot;:&quot;true&quot;,&quot;reason&quot;:&quot;&quot;}
  </code></pre>
  <p>• kubectl get nodes -o wide</p>
  <pre><code>NAME              STATUS   ROLES           AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
  in-dt-dckr-ldr1   Ready    control-plane   386d   v1.25.2   10.64.1.98    &lt;none&gt;        Ubuntu 20.04.5 LTS   5.15.0-58-generic   containerd://1.6.8
  in-dt-dckr-ldr2   Ready    &lt;none&gt;          386d   v1.25.2   10.64.1.190   &lt;none&gt;        Ubuntu 20.04.5 LTS   5.15.0-58-generic   containerd://1.6.8
  in-dt-dckr-wrk1   Ready    worker          386d   v1.25.2   10.64.1.232   &lt;none&gt;        Ubuntu 20.04.5 LTS   5.15.0-58-generic   containerd://1.6.8
  </code></pre>
  <p>•  kubectl describe node in-dt-dckr-wrk1</p>
  <pre><code>Name:               in-dt-dckr-wrk1
  Roles:              worker
  Labels:             beta.kubernetes.io/arch=amd64
                      beta.kubernetes.io/os=linux
                      kubernetes.io/arch=amd64
                      kubernetes.io/hostname=in-dt-dckr-wrk1
                      kubernetes.io/os=linux
                      node-role.kubernetes.io/worker=worker
  Annotations:        csi.volume.kubernetes.io/nodeid: {&quot;csi.tigera.io&quot;:&quot;in-dt-dckr-wrk1&quot;}
                      kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
                      node.alpha.kubernetes.io/ttl: 0
                      projectcalico.org/IPv4Address: 10.64.1.232/22
                      projectcalico.org/IPv4VXLANTunnelAddr: 192.168.137.192
                      volumes.kubernetes.io/controller-managed-attach-detach: true
  CreationTimestamp:  Thu, 13 Oct 2022 01:04:07 +0530
  Taints:             &lt;none&gt;
  Unschedulable:      false
  Lease:
    HolderIdentity:  in-dt-dckr-wrk1
    AcquireTime:     &lt;unset&gt;
    RenewTime:       Fri, 03 Nov 2023 08:57:17 +0530
  Conditions:
    Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
    ----                 ------  -----------------                 ------------------                ------                       -------
    NetworkUnavailable   False   Sat, 21 Jan 2023 19:29:25 +0530   Sat, 21 Jan 2023 19:29:25 +0530   CalicoIsUp                   Calico is running on this node
    MemoryPressure       False   Fri, 03 Nov 2023 08:53:02 +0530   Sat, 21 Jan 2023 19:28:44 +0530   KubeletHasSufficientMemory   kubelet has sufficient memory available
    DiskPressure         False   Fri, 03 Nov 2023 08:53:02 +0530   Sat, 21 Jan 2023 19:28:44 +0530   KubeletHasNoDiskPressure     kubelet has no disk pressure
    PIDPressure          False   Fri, 03 Nov 2023 08:53:02 +0530   Sat, 21 Jan 2023 19:28:44 +0530   KubeletHasSufficientPID      kubelet has sufficient PID available
    Ready                True    Fri, 03 Nov 2023 08:53:02 +0530   Sat, 21 Jan 2023 19:28:44 +0530   KubeletReady                 kubelet is posting ready status. AppArmor enabled
  Addresses:
    InternalIP:  10.64.1.232
    Hostname:    in-dt-dckr-wrk1
  Capacity:
    cpu:                8
    ephemeral-storage:  959786032Ki
    hugepages-2Mi:      0
    memory:             8023576Ki
    pods:               110
  Allocatable:
    cpu:                8
    ephemeral-storage:  884538805627
    hugepages-2Mi:      0
    memory:             7921176Ki
    pods:               110
  System Info:
    Machine ID:                 9e7f87693fba4ce5b5cffd58e27c4b5e
    System UUID:                4c4c4544-0032-5910-804e-b3c04f595831
    Boot ID:                    8c6f8a8c-0acc-4ac9-86df-36867df44d8f
    Kernel Version:             5.15.0-58-generic
    OS Image:                   Ubuntu 20.04.5 LTS
    Operating System:           linux
    Architecture:               amd64
    Container Runtime Version:  containerd://1.6.8
    Kubelet Version:            v1.25.2
    Kube-Proxy Version:         v1.25.2
  PodCIDR:                      192.168.1.0/24
  PodCIDRs:                     192.168.1.0/24
  Non-terminated Pods:          (18 in total)
    Namespace                   Name                                        CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
    ---------                   ----                                        ------------  ----------  ---------------  -------------  ---
    buildninja                  bn-agent1-6f484b8894-qr88p                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         52d
    buildninja                  bn-agent2-67b6859f58-zg6kc                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         52d
    calico-apiserver            calico-apiserver-978f5784c-77gss            0 (0%)        0 (0%)      0 (0%)           0 (0%)         346d
    calico-system               calico-kube-controllers-6fdfb4dfdc-rfbsr    0 (0%)        0 (0%)      0 (0%)           0 (0%)         346d
    calico-system               calico-node-4gxvn                           0 (0%)        0 (0%)      0 (0%)           0 (0%)         386d
    calico-system               calico-typha-5cd5cbc888-mcqfh               0 (0%)        0 (0%)      0 (0%)           0 (0%)         386d
    calico-system               csi-node-driver-zr5lc                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         386d
    db                          mongodb-cc47dc8b8-wwszq                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         178d
    db                          postgres-79f797b96c-vhswc                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         285d
    docker-registry             gci-docker-registry-644799b4b7-q8wbl        0 (0%)        0 (0%)      0 (0%)           0 (0%)         283d
    kube-system                 coredns-5fcc5bdd47-b6w88                    100m (1%)     0 (0%)      70Mi (0%)        170Mi (2%)     283d
    kube-system                 kube-proxy-zjp4z                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         386d
    kube-system                 metrics-server-5db9b4b966-8gqgk             100m (1%)     0 (0%)      200Mi (2%)       0 (0%)         386d
    tigera-operator             tigera-operator-6675dc47f4-z9jtx            0 (0%)        0 (0%)      0 (0%)           0 (0%)         386d
    utils                       web-sample-ff4d6596-hmtw6                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         170d
    utils                       wiki-server-86f49bd4c4-4drdr                0 (0%)        0 (0%)      0 (0%)           0 (0%)         182d
  Allocated resources:
    (Total limits may be over 100 percent, i.e., overcommitted.)
    Resource           Requests    Limits
    --------           --------    ------
    cpu                200m (2%)   0 (0%)
    memory             270Mi (3%)  170Mi (2%)
    ephemeral-storage  0 (0%)      0 (0%)
    hugepages-2Mi      0 (0%)      0 (0%)
  Events:              &lt;none&gt;
  </code></pre>
  <p>• kubectl describe po nginx</p>
  <pre><code>Name:             nginx
  Namespace:        default
  Priority:         0
  Service Account:  default
  Node:             &lt;none&gt;
  Labels:           run=nginx
  Annotations:      &lt;none&gt;
  Status:           Pending
  IP:
  IPs:              &lt;none&gt;
  Containers:
    nginx:
      Image:        nginx
      Port:         &lt;none&gt;
      Host Port:    &lt;none&gt;
      Environment:  &lt;none&gt;
      Mounts:
        /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lg45r (ro)
  Volumes:
    kube-api-access-lg45r:
      Type:                    Projected (a volume that contains injected data from multiple sources)
      TokenExpirationSeconds:  3607
      ConfigMapName:           kube-root-ca.crt
      ConfigMapOptional:       &lt;nil&gt;
      DownwardAPI:             true
  QoS Class:                   BestEffort
  Node-Selectors:              &lt;none&gt;
  Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                               node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  Events:                      &lt;none&gt;
  </code></pre>
  <p>I found a similar case but no resolutions there too
  <a href="https://stackoverflow.com/questions/55310076/pods-are-in-pending-state">https://stackoverflow.com/questions/55310076/pods-are-in-pending-state</a></p>
  <p><strong>Update 1</strong>
  Adding output for journalctl</p>
  <p>•  sudo journalctl -u kubelet -f</p>
  <pre><code>-- Logs begin at Fri 2022-04-22 10:32:53 IST. --
  Nov 01 14:18:45 IN-DT-DCKR-LDR1 kubelet[3279607]: I1101 14:18:45.802594 3279607 request.go:682] Waited for 1.137588349s due to client-side throttling, not priority and fairness, request: POST:https://10.64.1.98:6443/api/v1/namespaces/kube-system/pods
  Nov 01 14:18:45 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:45.845718 3279607 kubelet.go:1712] &quot;Failed creating a mirror pod for&quot; err=&quot;pods \&quot;etcd-in-dt-dckr-ldr1\&quot; already exists&quot; pod=&quot;kube-system/etcd-in-dt-dckr-ldr1&quot;
  Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.123742 3279607 configmap.go:197] Couldn't get configMap kube-system/kube-proxy: failed to sync configmap cache: timed out waiting for the condition
  Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.123782 3279607 configmap.go:197] Couldn't get configMap calico-system/tigera-ca-bundle: failed to sync configmap cache: timed out waiting for the condition
  Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.123870 3279607 nestedpendingoperations.go:348] Operation for &quot;{volumeName:kubernetes.io/configmap/61a2bf9a-a4cb-49d8-8b63-de7a82841a55-kube-proxy podName:61a2bf9a-a4cb-49d8-8b63-de7a82841a55 nodeName:}&quot; failed. No retries permitted until 2023-11-01 14:18:46.62383261 +0530 IST m=+3.132668643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume &quot;kube-proxy&quot; (UniqueName: &quot;kubernetes.io/configmap/61a2bf9a-a4cb-49d8-8b63-de7a82841a55-kube-proxy&quot;) pod &quot;kube-proxy-6ghjb&quot; (UID: &quot;61a2bf9a-a4cb-49d8-8b63-de7a82841a55&quot;) : failed to sync configmap cache: timed out waiting for the condition
  Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.123907 3279607 nestedpendingoperations.go:348] Operation for &quot;{volumeName:kubernetes.io/configmap/43736256-5241-46c4-b403-85db6d1d4d52-tigera-ca-bundle podName:43736256-5241-46c4-b403-85db6d1d4d52 nodeName:}&quot; failed. No retries permitted until 2023-11-01 14:18:46.623887546 +0530 IST m=+3.132723570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume &quot;tigera-ca-bundle&quot; (UniqueName: &quot;kubernetes.io/configmap/43736256-5241-46c4-b403-85db6d1d4d52-tigera-ca-bundle&quot;) pod &quot;calico-node-rb5s8&quot; (UID: &quot;43736256-5241-46c4-b403-85db6d1d4d52&quot;) : failed to sync configmap cache: timed out waiting for the condition
  Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.124885 3279607 configmap.go:197] Couldn't get configMap kube-system/coredns: failed to sync configmap cache: timed out waiting for the conditionNov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.124929 3279607 secret.go:192] Couldn't get secret calico-system/node-certs: failed to sync secret cache: timed out waiting for the condition
  Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.125014 3279607 nestedpendingoperations.go:348] Operation for &quot;{volumeName:kubernetes.io/configmap/08f1a4fd-92c5-48fc-a299-636a4f0a469a-config-volume podName:08f1a4fd-92c5-48fc-a299-636a4f0a469a nodeName:}&quot; failed. No retries permitted until 2023-11-01 14:18:46.624972044 +0530 IST m=+3.133808093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume &quot;config-volume&quot; (UniqueName: &quot;kubernetes.io/configmap/08f1a4fd-92c5-48fc-a299-636a4f0a469a-config-volume&quot;) pod &quot;coredns-5fcc5bdd47-8llkc&quot; (UID: &quot;08f1a4fd-92c5-48fc-a299-636a4f0a469a&quot;) : failed to sync configmap cache: timed out waiting for the condition
  Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.125058 3279607 nestedpendingoperations.go:348] Operation for &quot;{volumeName:kubernetes.io/secret/43736256-5241-46c4-b403-85db6d1d4d52-node-certs podName:43736256-5241-46c4-b403-85db6d1d4d52 nodeName:}&quot; failed. No retries permitted until 2023-11-01 14:18:46.625034355 +0530 IST m=+3.133870397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume &quot;node-certs&quot; (UniqueName: &quot;kubernetes.io/secret/43736256-5241-46c4-b403-85db6d1d4d52-node-certs&quot;) pod &quot;calico-node-rb5s8&quot; (UID: &quot;43736256-5241-46c4-b403-85db6d1d4d52&quot;) : failed to sync secret cache: timed out waiting for the condition
  </code></pre>
  <p>• sudo journalctl -u containerd -f</p>
  <pre><code>-- Logs begin at Fri 2022-04-22 10:32:53 IST. --
  Oct 13 00:13:20 IN-DT-DCKR-LDR1 containerd[828]: time=&quot;2023-10-13T00:13:20.769518216+05:30&quot; level=info msg=&quot;RemovePodSandbox for \&quot;ea81a51660843963fda52520e4d5834bb065e8bc6b48d7350fea5ed462066936\&quot;&quot;
  Oct 13 00:13:20 IN-DT-DCKR-LDR1 containerd[828]: time=&quot;2023-10-13T00:13:20.769565086+05:30&quot; level=info msg=&quot;Forcibly stopping sandbox \&quot;ea81a51660843963fda52520e4d5834bb065e8bc6b48d7350fea5ed462066936\&quot;&quot;
  Oct 13 00:13:20 IN-DT-DCKR-LDR1 containerd[828]: time=&quot;2023-10-13T00:13:20.769696525+05:30&quot; level=info msg=&quot;TearDown network for sandbox \&quot;ea81a51660843963fda52520e4d5834bb065e8bc6b48d7350fea5ed462066936\&quot; successfully&quot;
  Oct 13 00:13:20 IN-DT-DCKR-LDR1 containerd[828]: time=&quot;2023-10-13T00:13:20.852087515+05:30&quot; level=info msg=&quot;RemovePodSandbox \&quot;ea81a51660843963fda52520e4d5834bb065e8bc6b48d7350fea5ed462066936\&quot; returns successfully&quot;
  Nov 01 12:16:31 IN-DT-DCKR-LDR1 containerd[828]: time=&quot;2023-11-01T12:16:31.730407989+05:30&quot; level=info msg=&quot;No cni config template is specified, wait for other system components to drop the config.&quot;
  Nov 01 14:18:43 IN-DT-DCKR-LDR1 containerd[828]: time=&quot;2023-11-01T14:18:43.710560884+05:30&quot; level=info msg=&quot;No cni config template is specified, wait for other system components to drop the config.&quot;
  Nov 01 14:43:42 IN-DT-DCKR-LDR1 containerd[828]: E1101 14:43:42.533234     828 exec.go:87] error executing command in container: failed to exec in container: failed to start exec &quot;845f8446d19b7636bb87b001b881ce7b1f07dc372e61b542dd316605313e5bff&quot;: OCI runtime exec failed: exec failed: unable to start container process: exec: &quot;bash&quot;: executable file not found in $PATH: unknown
  Nov 01 14:44:30 IN-DT-DCKR-LDR1 containerd[828]: time=&quot;2023-11-01T14:44:30.136995113+05:30&quot; level=info msg=&quot;Container exec \&quot;b7fb9daf0322fbdbe4facb31c3d18fdd44d4c6402dc7733dbf5b2c4d659dcb72\&quot; stdin closed&quot;
  Nov 01 14:44:30 IN-DT-DCKR-LDR1 containerd[828]: E1101 14:44:30.288136     828 exec.go:87] error executing command in container: failed to exec in container: failed to start exec &quot;89067c1007468b006bc9b8d86b446ebbd04438025dd42cd90828123dc1a688b7&quot;: OCI runtime exec failed: exec failed: unable to start container process: exec: &quot;powershell&quot;: executable file not found in $PATH: unknown
  Nov 01 14:44:30 IN-DT-DCKR-LDR1 containerd[828]: E1101 14:44:30.419019     828 exec.go:87] error executing command in container: failed to exec in container: failed to start exec &quot;370d225ba5a9bc40bdc43040b1eda910fe278bd0bf5174682a94bc63a3be18ce&quot;: OCI runtime exec failed: exec failed: unable to start container process: exec: &quot;cmd&quot;: executable file not found in $PATH: unknown
  </code></pre>
  
Tags: <kubernetes><kubectl><kubeadm>
Title: Kubenetes new Pods/Deployments/Resources stuck in pending state forever without error
UserDisplayName: Sharad Tomer
ThreadText: # Kubenetes new Pods/Deployments/Resources stuck in pending state forever without error
  
  (Asked by: username_0 on 2023-11-03)
  
  I have a local Kubernetes cluster setup using kubeadm. Everything was working fine a few days back but now I've got a strange problem.
  
  Whenever I add new resources be it a deployment or pod, it never gets scheduled and is forever stuck in Pending state without any error. Please let me know if you anyone has any idea how to fix this issue or even debug for that matter.
  
  I'm pasting output of some of the commands I think might be useful, let me know if you need any additional information
  
  • kubectl get cs
  
  ```
  NAME                 STATUS    MESSAGE                         ERROR
  scheduler            Healthy   ok
  controller-manager   Healthy   ok
  etcd-0               Healthy   {"health":"true","reason":""}
  
  ```
  
  • kubectl get nodes -o wide
  
  ```
  NAME              STATUS   ROLES           AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
  in-dt-dckr-ldr1   Ready    control-plane   386d   v1.25.2   10.64.1.98    <none>        Ubuntu 20.04.5 LTS   5.15.0-58-generic   containerd://1.6.8
  in-dt-dckr-ldr2   Ready    <none>          386d   v1.25.2   10.64.1.190   <none>        Ubuntu 20.04.5 LTS   5.15.0-58-generic   containerd://1.6.8
  in-dt-dckr-wrk1   Ready    worker          386d   v1.25.2   10.64.1.232   <none>        Ubuntu 20.04.5 LTS   5.15.0-58-generic   containerd://1.6.8
  
  ```
  
  • kubectl describe node in-dt-dckr-wrk1
  
  ```
  Name:               in-dt-dckr-wrk1
  Roles:              worker
  Labels:             beta.kubernetes.io/arch=amd64
                      beta.kubernetes.io/os=linux
                      kubernetes.io/arch=amd64
                      kubernetes.io/hostname=in-dt-dckr-wrk1
                      kubernetes.io/os=linux
                      node-role.kubernetes.io/worker=worker
  Annotations:        csi.volume.kubernetes.io/nodeid: {"csi.tigera.io":"in-dt-dckr-wrk1"}
                      kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
                      node.alpha.kubernetes.io/ttl: 0
                      projectcalico.org/IPv4Address: 10.64.1.232/22
                      projectcalico.org/IPv4VXLANTunnelAddr: 192.168.137.192
                      volumes.kubernetes.io/controller-managed-attach-detach: true
  CreationTimestamp:  Thu, 13 Oct 2022 01:04:07 +0530
  Taints:             <none>
  Unschedulable:      false
  Lease:
    HolderIdentity:  in-dt-dckr-wrk1
    AcquireTime:     <unset>
    RenewTime:       Fri, 03 Nov 2023 08:57:17 +0530
  Conditions:
    Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
    ----                 ------  -----------------                 ------------------                ------                       -------
    NetworkUnavailable   False   Sat, 21 Jan 2023 19:29:25 +0530   Sat, 21 Jan 2023 19:29:25 +0530   CalicoIsUp                   Calico is running on this node
    MemoryPressure       False   Fri, 03 Nov 2023 08:53:02 +0530   Sat, 21 Jan 2023 19:28:44 +0530   KubeletHasSufficientMemory   kubelet has sufficient memory available
    DiskPressure         False   Fri, 03 Nov 2023 08:53:02 +0530   Sat, 21 Jan 2023 19:28:44 +0530   KubeletHasNoDiskPressure     kubelet has no disk pressure
    PIDPressure          False   Fri, 03 Nov 2023 08:53:02 +0530   Sat, 21 Jan 2023 19:28:44 +0530   KubeletHasSufficientPID      kubelet has sufficient PID available
    Ready                True    Fri, 03 Nov 2023 08:53:02 +0530   Sat, 21 Jan 2023 19:28:44 +0530   KubeletReady                 kubelet is posting ready status. AppArmor enabled
  Addresses:
    InternalIP:  10.64.1.232
    Hostname:    in-dt-dckr-wrk1
  Capacity:
    cpu:                8
    ephemeral-storage:  959786032Ki
    hugepages-2Mi:      0
    memory:             8023576Ki
    pods:               110
  Allocatable:
    cpu:                8
    ephemeral-storage:  884538805627
    hugepages-2Mi:      0
    memory:             7921176Ki
    pods:               110
  System Info:
    Machine ID:                 9e7f87693fba4ce5b5cffd58e27c4b5e
    System UUID:                4c4c4544-0032-5910-804e-b3c04f595831
    Boot ID:                    8c6f8a8c-0acc-4ac9-86df-36867df44d8f
    Kernel Version:             5.15.0-58-generic
    OS Image:                   Ubuntu 20.04.5 LTS
    Operating System:           linux
    Architecture:               amd64
    Container Runtime Version:  containerd://1.6.8
    Kubelet Version:            v1.25.2
    Kube-Proxy Version:         v1.25.2
  PodCIDR:                      192.168.1.0/24
  PodCIDRs:                     192.168.1.0/24
  Non-terminated Pods:          (18 in total)
    Namespace                   Name                                        CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
    ---------                   ----                                        ------------  ----------  ---------------  -------------  ---
    buildninja                  bn-agent1-6f484b8894-qr88p                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         52d
    buildninja                  bn-agent2-67b6859f58-zg6kc                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         52d
    calico-apiserver            calico-apiserver-978f5784c-77gss            0 (0%)        0 (0%)      0 (0%)           0 (0%)         346d
    calico-system               calico-kube-controllers-6fdfb4dfdc-rfbsr    0 (0%)        0 (0%)      0 (0%)           0 (0%)         346d
    calico-system               calico-node-4gxvn                           0 (0%)        0 (0%)      0 (0%)           0 (0%)         386d
    calico-system               calico-typha-5cd5cbc888-mcqfh               0 (0%)        0 (0%)      0 (0%)           0 (0%)         386d
    calico-system               csi-node-driver-zr5lc                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         386d
    db                          mongodb-cc47dc8b8-wwszq                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         178d
    db                          postgres-79f797b96c-vhswc                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         285d
    docker-registry             gci-docker-registry-644799b4b7-q8wbl        0 (0%)        0 (0%)      0 (0%)           0 (0%)         283d
    kube-system                 coredns-5fcc5bdd47-b6w88                    100m (1%)     0 (0%)      70Mi (0%)        170Mi (2%)     283d
    kube-system                 kube-proxy-zjp4z                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         386d
    kube-system                 metrics-server-5db9b4b966-8gqgk             100m (1%)     0 (0%)      200Mi (2%)       0 (0%)         386d
    tigera-operator             tigera-operator-6675dc47f4-z9jtx            0 (0%)        0 (0%)      0 (0%)           0 (0%)         386d
    utils                       web-sample-ff4d6596-hmtw6                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         170d
    utils                       wiki-server-86f49bd4c4-4drdr                0 (0%)        0 (0%)      0 (0%)           0 (0%)         182d
  Allocated resources:
    (Total limits may be over 100 percent, i.e., overcommitted.)
    Resource           Requests    Limits
    --------           --------    ------
    cpu                200m (2%)   0 (0%)
    memory             270Mi (3%)  170Mi (2%)
    ephemeral-storage  0 (0%)      0 (0%)
    hugepages-2Mi      0 (0%)      0 (0%)
  Events:              <none>
  
  ```
  
  • kubectl describe po nginx
  
  ```
  Name:             nginx
  Namespace:        default
  Priority:         0
  Service Account:  default
  Node:             <none>
  Labels:           run=nginx
  Annotations:      <none>
  Status:           Pending
  IP:
  IPs:              <none>
  Containers:
    nginx:
      Image:        nginx
      Port:         <none>
      Host Port:    <none>
      Environment:  <none>
      Mounts:
        /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lg45r (ro)
  Volumes:
    kube-api-access-lg45r:
      Type:                    Projected (a volume that contains injected data from multiple sources)
      TokenExpirationSeconds:  3607
      ConfigMapName:           kube-root-ca.crt
      ConfigMapOptional:       <nil>
      DownwardAPI:             true
  QoS Class:                   BestEffort
  Node-Selectors:              <none>
  Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                               node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  Events:                      <none>
  
  ```
  
  I found a similar case but no resolutions there too
  <https://stackoverflow.com/questions/55310076/pods-are-in-pending-state>
  
  **Update 1**
  Adding output for journalctl
  
  • sudo journalctl -u kubelet -f
  
  ```
  -- Logs begin at Fri 2022-04-22 10:32:53 IST. --
  Nov 01 14:18:45 IN-DT-DCKR-LDR1 kubelet[3279607]: I1101 14:18:45.802594 3279607 request.go:682] Waited for 1.137588349s due to client-side throttling, not priority and fairness, request: POST:https://10.64.1.98:6443/api/v1/namespaces/kube-system/pods
  Nov 01 14:18:45 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:45.845718 3279607 kubelet.go:1712] "Failed creating a mirror pod for" err="pods \"etcd-in-dt-dckr-ldr1\" already exists" pod="kube-system/etcd-in-dt-dckr-ldr1"
  Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.123742 3279607 configmap.go:197] Couldn't get configMap kube-system/kube-proxy: failed to sync configmap cache: timed out waiting for the condition
  Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.123782 3279607 configmap.go:197] Couldn't get configMap calico-system/tigera-ca-bundle: failed to sync configmap cache: timed out waiting for the condition
  Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.123870 3279607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61a2bf9a-a4cb-49d8-8b63-de7a82841a55-kube-proxy podName:61a2bf9a-a4cb-49d8-8b63-de7a82841a55 nodeName:}" failed. No retries permitted until 2023-11-01 14:18:46.62383261 +0530 IST m=+3.132668643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/61a2bf9a-a4cb-49d8-8b63-de7a82841a55-kube-proxy") pod "kube-proxy-6ghjb" (UID: "61a2bf9a-a4cb-49d8-8b63-de7a82841a55") : failed to sync configmap cache: timed out waiting for the condition
  Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.123907 3279607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43736256-5241-46c4-b403-85db6d1d4d52-tigera-ca-bundle podName:43736256-5241-46c4-b403-85db6d1d4d52 nodeName:}" failed. No retries permitted until 2023-11-01 14:18:46.623887546 +0530 IST m=+3.132723570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tigera-ca-bundle" (UniqueName: "kubernetes.io/configmap/43736256-5241-46c4-b403-85db6d1d4d52-tigera-ca-bundle") pod "calico-node-rb5s8" (UID: "43736256-5241-46c4-b403-85db6d1d4d52") : failed to sync configmap cache: timed out waiting for the condition
  Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.124885 3279607 configmap.go:197] Couldn't get configMap kube-system/coredns: failed to sync configmap cache: timed out waiting for the conditionNov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.124929 3279607 secret.go:192] Couldn't get secret calico-system/node-certs: failed to sync secret cache: timed out waiting for the condition
  Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.125014 3279607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08f1a4fd-92c5-48fc-a299-636a4f0a469a-config-volume podName:08f1a4fd-92c5-48fc-a299-636a4f0a469a nodeName:}" failed. No retries permitted until 2023-11-01 14:18:46.624972044 +0530 IST m=+3.133808093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "config-volume" (UniqueName: "kubernetes.io/configmap/08f1a4fd-92c5-48fc-a299-636a4f0a469a-config-volume") pod "coredns-5fcc5bdd47-8llkc" (UID: "08f1a4fd-92c5-48fc-a299-636a4f0a469a") : failed to sync configmap cache: timed out waiting for the condition
  Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.125058 3279607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43736256-5241-46c4-b403-85db6d1d4d52-node-certs podName:43736256-5241-46c4-b403-85db6d1d4d52 nodeName:}" failed. No retries permitted until 2023-11-01 14:18:46.625034355 +0530 IST m=+3.133870397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-certs" (UniqueName: "kubernetes.io/secret/43736256-5241-46c4-b403-85db6d1d4d52-node-certs") pod "calico-node-rb5s8" (UID: "43736256-5241-46c4-b403-85db6d1d4d52") : failed to sync secret cache: timed out waiting for the condition
  
  ```
  
  • sudo journalctl -u containerd -f
  
  ```
  -- Logs begin at Fri 2022-04-22 10:32:53 IST. --
  Oct 13 00:13:20 IN-DT-DCKR-LDR1 containerd[828]: time="2023-10-13T00:13:20.769518216+05:30" level=info msg="RemovePodSandbox for \"ea81a51660843963fda52520e4d5834bb065e8bc6b48d7350fea5ed462066936\""
  Oct 13 00:13:20 IN-DT-DCKR-LDR1 containerd[828]: time="2023-10-13T00:13:20.769565086+05:30" level=info msg="Forcibly stopping sandbox \"ea81a51660843963fda52520e4d5834bb065e8bc6b48d7350fea5ed462066936\""
  Oct 13 00:13:20 IN-DT-DCKR-LDR1 containerd[828]: time="2023-10-13T00:13:20.769696525+05:30" level=info msg="TearDown network for sandbox \"ea81a51660843963fda52520e4d5834bb065e8bc6b48d7350fea5ed462066936\" successfully"
  Oct 13 00:13:20 IN-DT-DCKR-LDR1 containerd[828]: time="2023-10-13T00:13:20.852087515+05:30" level=info msg="RemovePodSandbox \"ea81a51660843963fda52520e4d5834bb065e8bc6b48d7350fea5ed462066936\" returns successfully"
  Nov 01 12:16:31 IN-DT-DCKR-LDR1 containerd[828]: time="2023-11-01T12:16:31.730407989+05:30" level=info msg="No cni config template is specified, wait for other system components to drop the config."
  Nov 01 14:18:43 IN-DT-DCKR-LDR1 containerd[828]: time="2023-11-01T14:18:43.710560884+05:30" level=info msg="No cni config template is specified, wait for other system components to drop the config."
  Nov 01 14:43:42 IN-DT-DCKR-LDR1 containerd[828]: E1101 14:43:42.533234     828 exec.go:87] error executing command in container: failed to exec in container: failed to start exec "845f8446d19b7636bb87b001b881ce7b1f07dc372e61b542dd316605313e5bff": OCI runtime exec failed: exec failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown
  Nov 01 14:44:30 IN-DT-DCKR-LDR1 containerd[828]: time="2023-11-01T14:44:30.136995113+05:30" level=info msg="Container exec \"b7fb9daf0322fbdbe4facb31c3d18fdd44d4c6402dc7733dbf5b2c4d659dcb72\" stdin closed"
  Nov 01 14:44:30 IN-DT-DCKR-LDR1 containerd[828]: E1101 14:44:30.288136     828 exec.go:87] error executing command in container: failed to exec in container: failed to start exec "89067c1007468b006bc9b8d86b446ebbd04438025dd42cd90828123dc1a688b7": OCI runtime exec failed: exec failed: unable to start container process: exec: "powershell": executable file not found in $PATH: unknown
  Nov 01 14:44:30 IN-DT-DCKR-LDR1 containerd[828]: E1101 14:44:30.419019     828 exec.go:87] error executing command in container: failed to exec in container: failed to start exec "370d225ba5a9bc40bdc43040b1eda910fe278bd0bf5174682a94bc63a3be18ce": OCI runtime exec failed: exec failed: unable to start container process: exec: "cmd": executable file not found in $PATH: unknown
  
  ```
  
  **Tags:** kubernetes, kubectl, kubeadm
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-03. Score: 0
  
  You will probably find the error in either
  
  ```
  sudo journalctl -u kubelet -f
  
  ```
  
  or
  
  ```
  sudo journalctl -u containerd -f
  
  ```
  
  My guess would be that you have SystemdCgroup set to false in /etc/containerd/config.toml
  
  Set it to true on all nodes and restart containerd
  
  ```
  SystemdCgroup = true
  
  ```
  
  when you dont have the file yet, create it
  
  `containerd config default > /etc/containerd/config.toml`
  
  then set SystemdCgroup to true and restart containerd
  
  > **Comments:**
  > **username_0**: thank you,  I checked the journalctl and found that there is some error related to 'configmap cache sync'. I've added the output of both the command in the question, do you have any idea about the same? Also regarding, config.toml file, there is no such file in my system,  
  > **username_1**: edited the answer to create congig.toml. you are using calico for CNI, try use flannel if the containerd config solution does not work. flannel is easier to begin with but it lacks some features  
  
  ---
  
  