<p dir="auto">Kubelet relies on <code class="notranslate">docker history</code> to get the size of layers for each image, so that it can deduplicate shared layers. This is fine for docker v1.9 or before. For docker 1.10+, docker stops generating a random image ID for each layer, and just uses the content addressable IDs. Except for the top layer, every other layer has a <code class="notranslate">&lt;missing&gt;</code> ID. Kubelet relies on using the image layer IDs as keys, and this affects the calculation of total used space.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="======== d023c050c065 ===========
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
d023c050c065        7 days ago          /bin/sh -c #(nop) ENTRYPOINT &amp;{[&quot;/dashboard&quot;    0 B                 
&lt;missing&gt;           7 days ago          /bin/sh -c #(nop) EXPOSE 9090/tcp               0 B                 
&lt;missing&gt;           7 days ago          /bin/sh -c #(nop) ADD dir:6bf26a2727be5df090a   58.65 MB            
&lt;missing&gt;           4 months ago        /bin/sh -c #(nop) MAINTAINER Piotr Bryk &lt;bryk   0 B                 
======== 18ea23a675da ===========
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
18ea23a675da        2 weeks ago         /bin/sh -c #(nop) CMD [&quot;nginx&quot; &quot;-g&quot; &quot;daemon o   0 B                 
&lt;missing&gt;           2 weeks ago         /bin/sh -c #(nop) EXPOSE 443/tcp 80/tcp         0 B                 
&lt;missing&gt;           2 weeks ago         /bin/sh -c ln -sf /dev/stderr /var/log/nginx/   11 B                
&lt;missing&gt;           2 weeks ago         /bin/sh -c ln -sf /dev/stdout /var/log/nginx/   11 B                
&lt;missing&gt;           2 weeks ago         /bin/sh -c /tmp/build.sh                        53.91 MB            
&lt;missing&gt;           2 weeks ago         /bin/sh -c #(nop) COPY file:bf2b48e7f6cbf1964   8.221 kB            
&lt;missing&gt;           7 weeks ago         /bin/sh -c #(nop) CMD [&quot;/bin/sh&quot;]               0 B                 
&lt;missing&gt;           7 weeks ago         /bin/sh -c #(nop) ADD file:2df43ffb0761746c1f   56.57 MB            "><pre class="notranslate"><code class="notranslate">======== d023c050c065 ===========
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
d023c050c065        7 days ago          /bin/sh -c #(nop) ENTRYPOINT &amp;{["/dashboard"    0 B                 
&lt;missing&gt;           7 days ago          /bin/sh -c #(nop) EXPOSE 9090/tcp               0 B                 
&lt;missing&gt;           7 days ago          /bin/sh -c #(nop) ADD dir:6bf26a2727be5df090a   58.65 MB            
&lt;missing&gt;           4 months ago        /bin/sh -c #(nop) MAINTAINER Piotr Bryk &lt;bryk   0 B                 
======== 18ea23a675da ===========
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
18ea23a675da        2 weeks ago         /bin/sh -c #(nop) CMD ["nginx" "-g" "daemon o   0 B                 
&lt;missing&gt;           2 weeks ago         /bin/sh -c #(nop) EXPOSE 443/tcp 80/tcp         0 B                 
&lt;missing&gt;           2 weeks ago         /bin/sh -c ln -sf /dev/stderr /var/log/nginx/   11 B                
&lt;missing&gt;           2 weeks ago         /bin/sh -c ln -sf /dev/stdout /var/log/nginx/   11 B                
&lt;missing&gt;           2 weeks ago         /bin/sh -c /tmp/build.sh                        53.91 MB            
&lt;missing&gt;           2 weeks ago         /bin/sh -c #(nop) COPY file:bf2b48e7f6cbf1964   8.221 kB            
&lt;missing&gt;           7 weeks ago         /bin/sh -c #(nop) CMD ["/bin/sh"]               0 B                 
&lt;missing&gt;           7 weeks ago         /bin/sh -c #(nop) ADD file:2df43ffb0761746c1f   56.57 MB            
</code></pre></div>
<p dir="auto">On the other hand, we can get all the layers from inspecting the image, but there is no information about its sizes.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="        &quot;RootFS&quot;: {
            &quot;Type&quot;: &quot;layers&quot;,
            &quot;Layers&quot;: [
                &quot;sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef&quot;,
                &quot;sha256:4636ee4859c26ceb06787821cbfb166a81ca82db61656ab01ae168d5e7dee664&quot;,
                &quot;sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef&quot;,
                &quot;sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef&quot;
            ]
        }"><pre class="notranslate"><code class="notranslate">        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
                "sha256:4636ee4859c26ceb06787821cbfb166a81ca82db61656ab01ae168d5e7dee664",
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
            ]
        }
</code></pre></div>
<p dir="auto">/cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/vishh/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/vishh">@vishh</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ronnielai/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ronnielai">@ronnielai</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/Random-Liu/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/Random-Liu">@Random-Liu</a></p>
<ol dir="auto">
<li>
<p dir="auto">Followed the README.md for cluster setup with 4 nodes using</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl scale rc cassandra --replicas=4"><pre class="notranslate"><code class="notranslate">$ kubectl scale rc cassandra --replicas=4
</code></pre></div>
</li>
<li>
<p dir="auto">Following status command supposed to list all 4-nodes in cluster</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl exec -ti cassandra -- nodetool status"><pre class="notranslate"><code class="notranslate">$ kubectl exec -ti cassandra -- nodetool status
</code></pre></div>
</li>
</ol>
<p dir="auto">Instead of 4-nodes, we <strong>see just 1-node</strong>.</p>
<p dir="auto">Seems like the "KubernetesSeedProvider" is not doing the needed cluster discovery?</p>