Answers: <p>I think I fixed the issue.</p>
  <p>It seems it comes up, when the <code>kubelet</code> service starts before the <code>docker</code> service is up and running. So I added the directive <code>After=docker.service</code> to the <code>[Unit]</code> block of the <code>kubelet</code> service and restarted it. The issue seems to be gone now.</p>
  
Answers: CC BY-SA 4.0
Answers: 2022-09-04T14:53:55.593Z
Answers: 2022-09-04T14:53:55.593Z
Answers: Max N.
Answers: <p>Quick fix by below works for me:</p>
  <pre><code>docker system prune
  systemctl stop kubelet
  systemctl stop docker
  systemctl start docker
  systemctl start kubelet
  </code></pre>
  
Answers: CC BY-SA 4.0
Answers: 2023-05-21T18:31:05.970Z
Answers: Yep, this should work as long as the node is not restarted. Adding the `After` directive to the systemd service makes sure, that the services are always started in the correct order.
Answers: Max N.
Answers: CC BY-SA 4.0
Answers: 2023-05-18T01:27:44.060Z
Answers: 2023-05-18T01:27:44.060Z
Answers: xiaojueguan
Body: <p>For quiet some time every once in a while I get an alert, that the disk of one of my Kubernetes cluster nodes is filling up. I pretty quickly found out, that it were the Docker images. As I did not have time to deeper analyze the issue, I just used <code>docker system prune</code> to get rid of everything not running (yeah, I know this is highly discouraged).</p>
  <p>Now I started to look into this issue and it seems, that the garbage collector has issues with the images:</p>
  <pre class="lang-bash prettyprint-override"><code>max@nb [~]
  -&gt; % k get events -owide
  LAST SEEN   TYPE      REASON          OBJECT          SUBOBJECT   SOURCE                MESSAGE                                                             FIRST SEEN   COUNT   NAME
  3m15s       Warning   ImageGCFailed   node/cluster00              kubelet, cluster00    failed to get imageFs info: non-existent label &quot;docker-images&quot;      69d          20073   cluster00.16fa70fb01b64a69
  4m15s       Warning   ImageGCFailed   node/cluster02              kubelet, cluster02    failed to get imageFs info: non-existent label &quot;docker-images&quot;      69d          20085   cluster02.16fa6da5f5b30c31
  54s         Warning   ImageGCFailed   node/cluster04              kubelet, cluster04    failed to get imageFs info: non-existent label &quot;docker-images&quot;      69d          20082   cluster04.16fa6ea64df694c5
  3m57s       Warning   ImageGCFailed   node/cluster05              kubelet, cluster05    failed to get imageFs info: non-existent label &quot;docker-images&quot;      69d          20087   cluster05.16fa6d1f167fe7f4
  48s         Warning   ImageGCFailed   node/cluster06              kubelet, cluster06    failed to get imageFs info: non-existent label &quot;docker-images&quot;      69d          20077   cluster06.16fa700540142542
  2m21s       Warning   ImageGCFailed   node/cluster11              kubelet, cluster11    failed to get imageFs info: non-existent label &quot;docker-images&quot;      69d          20074   cluster11.16fa70c17857fef8
  max@nb [~]
  -&gt; %
  </code></pre>
  <p>I have two other nodes in the cluster, that are newer than the others and do not seem to have the same issue:</p>
  <pre class="lang-bash prettyprint-override"><code>max@nb [~]
  -&gt; % kubectl get nodes
  NAME        STATUS   ROLES                  AGE      VERSION
  cluster00   Ready    control-plane,master   4y107d   v1.23.8
  cluster02   Ready    &lt;none&gt;                 2y270d   v1.23.8
  cluster04   Ready    &lt;none&gt;                 3y267d   v1.23.8
  cluster05   Ready    &lt;none&gt;                 4y107d   v1.23.8
  cluster06   Ready    &lt;none&gt;                 3y171d   v1.23.8
  cluster11   Ready    &lt;none&gt;                 592d     v1.23.8
  cluster12   Ready    &lt;none&gt;                 139d     v1.23.8
  cluster13   Ready    &lt;none&gt;                 139d     v1.23.8
  max@nb [~]
  -&gt; %
  </code></pre>
  <p>The cluster was initially setup more than four years ago manually with <code>kubeadm</code>. If I remember correctly, the version was 1.9.6 back then. And I updated it from there up to 1.23.8 right now. When the last two nodes (the working ones) were added, the cluster was on 1.19.8.</p>
  <p>It is very much possible, that I forgot some configuration change in any of the upgrades. 69 days ago (the time the first appeared as of <code>get events</code>) I did the upgrade from 1.19.8 to 1.23.8 (and all versions in-between, of course). Could be, that the issue started then. Could also be, that the upgrade <em>cleared</em> the events log. Not sure if the issue was already there before, to be honest.</p>
  <p>Has anyone any idea where I could start looking for the reason of this issue?</p>
  <p>Thanks in advance!</p>
  
Tags: <docker><kubernetes>
Title: Kubernetes fails to do do garbage collection on images
UserDisplayName: Max N.
ThreadText: # Kubernetes fails to do do garbage collection on images
  
  (Asked by: username_0 on 2022-08-30)
  
  For quiet some time every once in a while I get an alert, that the disk of one of my Kubernetes cluster nodes is filling up. I pretty quickly found out, that it were the Docker images. As I did not have time to deeper analyze the issue, I just used `docker system prune` to get rid of everything not running (yeah, I know this is highly discouraged).
  
  Now I started to look into this issue and it seems, that the garbage collector has issues with the images:
  
  ```bash
  max@nb [~]
  -> % k get events -owide
  LAST SEEN   TYPE      REASON          OBJECT          SUBOBJECT   SOURCE                MESSAGE                                                             FIRST SEEN   COUNT   NAME
  3m15s       Warning   ImageGCFailed   node/cluster00              kubelet, cluster00    failed to get imageFs info: non-existent label "docker-images"      69d          20073   cluster00.16fa70fb01b64a69
  4m15s       Warning   ImageGCFailed   node/cluster02              kubelet, cluster02    failed to get imageFs info: non-existent label "docker-images"      69d          20085   cluster02.16fa6da5f5b30c31
  54s         Warning   ImageGCFailed   node/cluster04              kubelet, cluster04    failed to get imageFs info: non-existent label "docker-images"      69d          20082   cluster04.16fa6ea64df694c5
  3m57s       Warning   ImageGCFailed   node/cluster05              kubelet, cluster05    failed to get imageFs info: non-existent label "docker-images"      69d          20087   cluster05.16fa6d1f167fe7f4
  48s         Warning   ImageGCFailed   node/cluster06              kubelet, cluster06    failed to get imageFs info: non-existent label "docker-images"      69d          20077   cluster06.16fa700540142542
  2m21s       Warning   ImageGCFailed   node/cluster11              kubelet, cluster11    failed to get imageFs info: non-existent label "docker-images"      69d          20074   cluster11.16fa70c17857fef8
  max@nb [~]
  -> %
  
  ```
  
  I have two other nodes in the cluster, that are newer than the others and do not seem to have the same issue:
  
  ```bash
  max@nb [~]
  -> % kubectl get nodes
  NAME        STATUS   ROLES                  AGE      VERSION
  cluster00   Ready    control-plane,master   4y107d   v1.23.8
  cluster02   Ready    <none>                 2y270d   v1.23.8
  cluster04   Ready    <none>                 3y267d   v1.23.8
  cluster05   Ready    <none>                 4y107d   v1.23.8
  cluster06   Ready    <none>                 3y171d   v1.23.8
  cluster11   Ready    <none>                 592d     v1.23.8
  cluster12   Ready    <none>                 139d     v1.23.8
  cluster13   Ready    <none>                 139d     v1.23.8
  max@nb [~]
  -> %
  
  ```
  
  The cluster was initially setup more than four years ago manually with `kubeadm`. If I remember correctly, the version was 1.9.6 back then. And I updated it from there up to 1.23.8 right now. When the last two nodes (the working ones) were added, the cluster was on 1.19.8.
  
  It is very much possible, that I forgot some configuration change in any of the upgrades. 69 days ago (the time the first appeared as of `get events`) I did the upgrade from 1.19.8 to 1.23.8 (and all versions in-between, of course). Could be, that the issue started then. Could also be, that the upgrade *cleared* the events log. Not sure if the issue was already there before, to be honest.
  
  Has anyone any idea where I could start looking for the reason of this issue?
  
  Thanks in advance!
  
  **Tags:** docker, kubernetes
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2022-09-04. Score: 1 | Accepted answer
  
  I think I fixed the issue.
  
  It seems it comes up, when the `kubelet` service starts before the `docker` service is up and running. So I added the directive `After=docker.service` to the `[Unit]` block of the `kubelet` service and restarted it. The issue seems to be gone now.
  
  
  ---
  
  ### Answer by username_1 on 2023-05-18. Score: 0
  
  Quick fix by below works for me:
  
  ```
  docker system prune
  systemctl stop kubelet
  systemctl stop docker
  systemctl start docker
  systemctl start kubelet
  
  ```
  
  > **Comments:**
  > **username_0**: Yep, this should work as long as the node is not restarted. Adding the `After` directive to the systemd service makes sure, that the services are always started in the correct order.  
  
  ---
  
  