Answers: <p>This is because of the <a href="https://devops.stackexchange.com/a/16085/18965">default settings for the eviction manager (currently 5% hard, and 10% minimum for image reclaim)</a> safeguard a new pod exhausting the space on your hard disk, and the metrics are a bit overly conservative. You can change the eviction manager settings by providing a <code>--kubelet-arg</code> that overrides the <a href="https://devops.stackexchange.com/a/16085/18965">k3s defaults.</a>. This can be persisted by providing <a href="https://devops.stackexchange.com/q/16070/18965"><code>kubelet-arg</code> to your k3s <code>KUBECONFIG</code> at <code>/etc/rancher/k3s/config.yaml</code></a>.</p>
  <p>Like this,</p>
  <pre class="lang-yaml prettyprint-override"><code>kubelet-arg:
    - &quot;eviction-minimum-reclaim=imagefs.available=2%,nodefs.available=2%&quot;
    - &quot;eviction-hard=memory.available&lt;500Mi,nodefs.available&lt;10Gi&quot;
  </code></pre>
  <hr />
  <p><a href="https://github.com/k3s-io/k3s/issues/5488#issuecomment-1147992468" rel="nofollow noreferrer">Note, while currently required, passing options by commandline to k3s is deprecated in upstream Kubernetes and is subject to removal in k3s when/if it gets finally removed.</a></p>
  
Answers: CC BY-SA 4.0
Answers: 2022-06-06T18:15:31.517Z
Answers: 2022-06-15T17:45:20.990Z
Answers: 2022-06-15T17:45:20.990Z
Answers: Evan Carroll
Body: <p>When I run <code>kubectl get pods -w --namespace kubeapps</code> I see all my pods are pending</p>
  <pre><code>NAME                                                         READY   STATUS    RESTARTS   AGE
  kubeapps-internal-dashboard-584c686cc6-qwjnf                 0/1     Pending   0          19h
  kubeapps-585bc584db-j994c                                    0/1     Pending   0          19h
  kubeapps-internal-apprepository-controller-5d7685c4f-zsxnk   0/1     Pending   0          19h
  kubeapps-internal-kubeops-7686579468-ddp8z                   0/1     Pending   0          19h
  kubeapps-internal-kubeappsapis-6f65bccf89-mzbtn              0/1     Pending   0          19h
  kubeapps-internal-dashboard-584c686cc6-wxnfd                 0/1     Pending   0          19h
  kubeapps-postgresql-0                                        0/1     Pending   0          19h
  kubeapps-585bc584db-rgvfq                                    0/1     Pending   0          19h
  kubeapps-internal-kubeops-7686579468-tzjwf                   0/1     Pending   0          19h
  kubeapps-internal-kubeappsapis-6f65bccf89-rbdrm              0/1     Pending   0          19h
  </code></pre>
  <p>When I check <code>journalctl -u k3s</code>, I see</p>
  <pre><code>E0604 11:37:07.861130    1301 configmap.go:200] Couldn't get configMap kube-system/chart-content-traefik: object &quot;kube-system&quot;/&quot;chart-content&gt;
  E0604 11:37:07.861184    1301 nestedpendingoperations.go:335] Operation for &quot;{volumeName:kubernetes.io/configmap/84367524-fc49-4db2-bf08-b217&gt;
  E0604 11:37:08.062023    1301 configmap.go:200] Couldn't get configMap kube-system/chart-content-traefik-crd: object &quot;kube-system&quot;/&quot;chart-con&gt;
  E0604 11:37:08.062042    1301 configmap.go:200] Couldn't get configMap kube-system/chart-values-traefik: object &quot;kube-system&quot;/&quot;chart-values-t&gt;
  E0604 11:37:08.062060    1301 configmap.go:200] Couldn't get configMap kube-system/chart-values-traefik-crd: object &quot;kube-system&quot;/&quot;chart-valu&gt;
  E0604 11:37:08.062089    1301 nestedpendingoperations.go:335] Operation for &quot;{volumeName:kubernetes.io/configmap/84367524-fc49-4db2-bf08-b217&gt;
  E0604 11:37:08.062099    1301 nestedpendingoperations.go:335] Operation for &quot;{volumeName:kubernetes.io/configmap/46bd29b8-1c81-4585-a8fb-ca7e&gt;
  E0604 11:37:08.062105    1301 nestedpendingoperations.go:335] Operation for &quot;{volumeName:kubernetes.io/configmap/46bd29b8-1c81-4585-a8fb-ca7e&gt;
  I0604 11:37:08.079010    1301 eviction_manager.go:338] &quot;Eviction manager: attempting to reclaim&quot; resourceName=&quot;ephemeral-storage&quot;
  I0604 11:37:08.079027    1301 container_gc.go:85] &quot;Attempting to delete unused containers&quot;
  I0604 11:37:08.079496    1301 image_gc_manager.go:327] &quot;Attempting to delete unused images&quot;
  I0604 11:37:08.082310    1301 eviction_manager.go:349] &quot;Eviction manager: must evict pod(s) to reclaim&quot; resourceName=&quot;ephemeral-storage&quot;
  I0604 11:37:08.082338    1301 eviction_manager.go:367] &quot;Eviction manager: pods ranked for eviction&quot; pods=[kube-system/coredns-d76bd69b-2svkn &gt;
  E0604 11:37:08.082353    1301 eviction_manager.go:560] &quot;Eviction manager: cannot evict a critical pod&quot; pod=&quot;kube-system/coredns-d76bd69b-2svk&gt;
  E0604 11:37:08.082359    1301 eviction_manager.go:560] &quot;Eviction manager: cannot evict a critical pod&quot; pod=&quot;kube-system/metrics-server-7cd5fc&gt;
  E0604 11:37:08.082363    1301 eviction_manager.go:560] &quot;Eviction manager: cannot evict a critical pod&quot; pod=&quot;kube-system/local-path-provisione&gt;
  I0604 11:37:08.082368    1301 eviction_manager.go:390] &quot;Eviction manager: unable to evict any pods from the node&quot;
  </code></pre>
  <p>I think this is a result of the eviction manager, because my system is <em>low</em> on disk space? Perhaps it's just looking at percent free, the below is the device from <code>df -h</code></p>
  <p>I'm also seeing this in the error log,</p>
  <pre><code>k3s[1301]: I0604 11:52:13.741674    1301 image_gc_manager.go:310] &quot;Disk usage on image filesystem is over the high threshold, trying to free bytes down to the low threshold&quot; usage=98 highThreshold=85 amountToFree=337223212236 lowThreshold=80
  k3s[1301]: E0604 11:52:13.742880    1301 kubelet.go:1347] &quot;Image garbage collection failed multiple times in a row&quot; err=&quot;failed to garbage collect required amount of images. Wanted to free 337223212236 bytes, but freed 0 bytes&quot;
  </code></pre>
  <pre><code>Filesystem                 Size  Used Avail Use% Mounted on
  /dev/mapper/x1c7--vg-root  1.8T  1.7T   50G  98% /
  </code></pre>
  <p>Where are these settings on k3s?</p>
  
Tags: <rancher><k3s><kubelet>
Title: k3s: Eviction manager: attempting to reclaim" resourceName="ephemeral-storage
UserDisplayName: Evan Carroll
ThreadText: # k3s: Eviction manager: attempting to reclaim" resourceName="ephemeral-storage
  
  (Asked by: username_0 on 2022-06-04)
  
  When I run `kubectl get pods -w --namespace kubeapps` I see all my pods are pending
  
  ```
  NAME                                                         READY   STATUS    RESTARTS   AGE
  kubeapps-internal-dashboard-584c686cc6-qwjnf                 0/1     Pending   0          19h
  kubeapps-585bc584db-j994c                                    0/1     Pending   0          19h
  kubeapps-internal-apprepository-controller-5d7685c4f-zsxnk   0/1     Pending   0          19h
  kubeapps-internal-kubeops-7686579468-ddp8z                   0/1     Pending   0          19h
  kubeapps-internal-kubeappsapis-6f65bccf89-mzbtn              0/1     Pending   0          19h
  kubeapps-internal-dashboard-584c686cc6-wxnfd                 0/1     Pending   0          19h
  kubeapps-postgresql-0                                        0/1     Pending   0          19h
  kubeapps-585bc584db-rgvfq                                    0/1     Pending   0          19h
  kubeapps-internal-kubeops-7686579468-tzjwf                   0/1     Pending   0          19h
  kubeapps-internal-kubeappsapis-6f65bccf89-rbdrm              0/1     Pending   0          19h
  
  ```
  
  When I check `journalctl -u k3s`, I see
  
  ```
  E0604 11:37:07.861130    1301 configmap.go:200] Couldn't get configMap kube-system/chart-content-traefik: object "kube-system"/"chart-content>
  E0604 11:37:07.861184    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/84367524-fc49-4db2-bf08-b217>
  E0604 11:37:08.062023    1301 configmap.go:200] Couldn't get configMap kube-system/chart-content-traefik-crd: object "kube-system"/"chart-con>
  E0604 11:37:08.062042    1301 configmap.go:200] Couldn't get configMap kube-system/chart-values-traefik: object "kube-system"/"chart-values-t>
  E0604 11:37:08.062060    1301 configmap.go:200] Couldn't get configMap kube-system/chart-values-traefik-crd: object "kube-system"/"chart-valu>
  E0604 11:37:08.062089    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/84367524-fc49-4db2-bf08-b217>
  E0604 11:37:08.062099    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/46bd29b8-1c81-4585-a8fb-ca7e>
  E0604 11:37:08.062105    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/46bd29b8-1c81-4585-a8fb-ca7e>
  I0604 11:37:08.079010    1301 eviction_manager.go:338] "Eviction manager: attempting to reclaim" resourceName="ephemeral-storage"
  I0604 11:37:08.079027    1301 container_gc.go:85] "Attempting to delete unused containers"
  I0604 11:37:08.079496    1301 image_gc_manager.go:327] "Attempting to delete unused images"
  I0604 11:37:08.082310    1301 eviction_manager.go:349] "Eviction manager: must evict pod(s) to reclaim" resourceName="ephemeral-storage"
  I0604 11:37:08.082338    1301 eviction_manager.go:367] "Eviction manager: pods ranked for eviction" pods=[kube-system/coredns-d76bd69b-2svkn >
  E0604 11:37:08.082353    1301 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/coredns-d76bd69b-2svk>
  E0604 11:37:08.082359    1301 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/metrics-server-7cd5fc>
  E0604 11:37:08.082363    1301 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/local-path-provisione>
  I0604 11:37:08.082368    1301 eviction_manager.go:390] "Eviction manager: unable to evict any pods from the node"
  
  ```
  
  I think this is a result of the eviction manager, because my system is *low* on disk space? Perhaps it's just looking at percent free, the below is the device from `df -h`
  
  I'm also seeing this in the error log,
  
  ```
  k3s[1301]: I0604 11:52:13.741674    1301 image_gc_manager.go:310] "Disk usage on image filesystem is over the high threshold, trying to free bytes down to the low threshold" usage=98 highThreshold=85 amountToFree=337223212236 lowThreshold=80
  k3s[1301]: E0604 11:52:13.742880    1301 kubelet.go:1347] "Image garbage collection failed multiple times in a row" err="failed to garbage collect required amount of images. Wanted to free 337223212236 bytes, but freed 0 bytes"
  
  ```
  
  ```
  Filesystem                 Size  Used Avail Use% Mounted on
  /dev/mapper/x1c7--vg-root  1.8T  1.7T   50G  98% /
  
  ```
  
  Where are these settings on k3s?
  
  **Tags:** rancher, k3s, kubelet
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-06-06. Score: 1 | Accepted answer
  
  This is because of the [default settings for the eviction manager (currently 5% hard, and 10% minimum for image reclaim)](https://devops.stackexchange.com/a/16085/18965) safeguard a new pod exhausting the space on your hard disk, and the metrics are a bit overly conservative. You can change the eviction manager settings by providing a `--kubelet-arg` that overrides the [k3s defaults.](https://devops.stackexchange.com/a/16085/18965). This can be persisted by providing [`kubelet-arg` to your k3s `KUBECONFIG` at `/etc/rancher/k3s/config.yaml`](https://devops.stackexchange.com/q/16070/18965).
  
  Like this,
  
  ```yaml
  kubelet-arg:
    - "eviction-minimum-reclaim=imagefs.available=2%,nodefs.available=2%"
    - "eviction-hard=memory.available<500Mi,nodefs.available<10Gi"
  
  ```
  
  ---
  
  [Note, while currently required, passing options by commandline to k3s is deprecated in upstream Kubernetes and is subject to removal in k3s when/if it gets finally removed.](https://github.com/k3s-io/k3s/issues/5488#issuecomment-1147992468)
  
  
  ---
  
  