<p dir="auto">When running <code class="notranslate">make test_e2e_node</code> on systemd, I get a single failure related to a test that verifies the stats endpoint.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="
------------------------------
• Failure [60.090 seconds]
Kubelet
/home/decarr/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:267
  metrics api
  /home/decarr/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:266
    when querying /stats/summary
    /home/decarr/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:258
      it should report resource usage through the stats api [It]
      /home/decarr/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:257

      Expected
          &lt;[]stats.ContainerStats | len:2, cap:4&gt;: ["><pre class="notranslate"><code class="notranslate">
------------------------------
• Failure [60.090 seconds]
Kubelet
/home/decarr/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:267
  metrics api
  /home/decarr/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:266
    when querying /stats/summary
    /home/decarr/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:258
      it should report resource usage through the stats api [It]
      /home/decarr/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:257

      Expected
          &lt;[]stats.ContainerStats | len:2, cap:4&gt;: [
</code></pre></div>
<p dir="auto">The root is this line:<br>
<a href="https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/kubelet_test.go#L223">https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/kubelet_test.go#L223</a></p>
<p dir="auto">On systemd, I am getting back 2 containers where 1 is expected from my stats endpoint.</p>
<p dir="auto">For example, if I run a pod with a single nginx container I get the following summary:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ curl http://127.0.0.1:10255/stats/summary
{
  &quot;node&quot;: {
   &quot;nodeName&quot;: &quot;127.0.0.1&quot;,
   &quot;systemContainers&quot;: [
    {
     &quot;name&quot;: &quot;kubelet&quot;,
     &quot;startTime&quot;: &quot;2016-03-07T14:17:36Z&quot;,
     &quot;cpu&quot;: {
      &quot;time&quot;: &quot;2016-03-07T17:55:11Z&quot;,
      &quot;usageNanoCores&quot;: 221608525,
      &quot;usageCoreNanoSeconds&quot;: 7174601461254
     },
     &quot;memory&quot;: {
      &quot;time&quot;: &quot;2016-03-07T17:55:11Z&quot;,
      &quot;usageBytes&quot;: 8132644864,
      &quot;workingSetBytes&quot;: 5518856192,
      &quot;pageFaults&quot;: 63072461,
      &quot;majorPageFaults&quot;: 10940
     },
     &quot;rootfs&quot;: {
      &quot;availableBytes&quot;: 18328821760,
      &quot;capacityBytes&quot;: 52710469632
     },
     &quot;logs&quot;: {
      &quot;availableBytes&quot;: 18328821760,
      &quot;capacityBytes&quot;: 52710469632
     },
     &quot;userDefinedMetrics&quot;: null
    },
    {
     &quot;name&quot;: &quot;runtime&quot;,
     &quot;startTime&quot;: &quot;2016-03-07T17:55:16Z&quot;,
     &quot;cpu&quot;: {
      &quot;time&quot;: &quot;2016-03-07T17:55:00Z&quot;,
      &quot;usageNanoCores&quot;: 17139366,
      &quot;usageCoreNanoSeconds&quot;: 213037971066
     },
     &quot;memory&quot;: {
      &quot;time&quot;: &quot;2016-03-07T17:55:00Z&quot;,
      &quot;usageBytes&quot;: 31969280,
      &quot;workingSetBytes&quot;: 25382912,
      &quot;pageFaults&quot;: 9133069,
      &quot;majorPageFaults&quot;: 14
     },
     &quot;rootfs&quot;: {
      &quot;availableBytes&quot;: 18328821760,
      &quot;capacityBytes&quot;: 52710469632
     },
     &quot;logs&quot;: {
      &quot;availableBytes&quot;: 18328821760,
      &quot;capacityBytes&quot;: 52710469632
     },
     &quot;userDefinedMetrics&quot;: null
    }
   ],
   &quot;startTime&quot;: &quot;2016-03-07T14:17:36Z&quot;,
   &quot;cpu&quot;: {
    &quot;time&quot;: &quot;2016-03-07T17:55:11Z&quot;,
    &quot;usageNanoCores&quot;: 221608525,
    &quot;usageCoreNanoSeconds&quot;: 7174601461254
   },
   &quot;memory&quot;: {
    &quot;time&quot;: &quot;2016-03-07T17:55:11Z&quot;,
    &quot;usageBytes&quot;: 8132644864,
    &quot;workingSetBytes&quot;: 5518856192,
    &quot;pageFaults&quot;: 63072461,
    &quot;majorPageFaults&quot;: 10940
   },
   &quot;fs&quot;: {
    &quot;availableBytes&quot;: 18328821760,
    &quot;capacityBytes&quot;: 52710469632,
    &quot;usedBytes&quot;: 31680516096
   }
  },
  &quot;pods&quot;: [
   {
    &quot;podRef&quot;: {
     &quot;name&quot;: &quot;nginx-2040093540-7uo6d&quot;,
     &quot;namespace&quot;: &quot;default&quot;,
     &quot;uid&quot;: &quot;a514f7cf-e48d-11e5-bd2c-28d2444e470d&quot;
    },
    &quot;startTime&quot;: &quot;2016-03-07T17:55:02Z&quot;,
    &quot;containers&quot;: [
     {
      &quot;name&quot;: &quot;nginx&quot;,
      &quot;startTime&quot;: &quot;2016-03-07T17:55:04Z&quot;,
      &quot;cpu&quot;: {
       &quot;time&quot;: &quot;2016-03-07T17:55:04Z&quot;,
       &quot;usageCoreNanoSeconds&quot;: 21626484
      },
      &quot;memory&quot;: {
       &quot;time&quot;: &quot;2016-03-07T17:55:04Z&quot;,
       &quot;usageBytes&quot;: 8437760,
       &quot;workingSetBytes&quot;: 2482176,
       &quot;pageFaults&quot;: 883,
       &quot;majorPageFaults&quot;: 48
      },
      &quot;rootfs&quot;: {
       &quot;availableBytes&quot;: 18328821760,
       &quot;capacityBytes&quot;: 52710469632
      },
      &quot;logs&quot;: {
       &quot;availableBytes&quot;: 18328821760,
       &quot;capacityBytes&quot;: 52710469632
      },
      &quot;userDefinedMetrics&quot;: null
     },
     {
      &quot;name&quot;: &quot;nginx&quot;,
      &quot;startTime&quot;: &quot;2016-03-07T17:55:04Z&quot;,
      &quot;cpu&quot;: {
       &quot;time&quot;: &quot;2016-03-07T17:55:04Z&quot;,
       &quot;usageCoreNanoSeconds&quot;: 0
      },
      &quot;memory&quot;: {
       &quot;time&quot;: &quot;2016-03-07T17:55:04Z&quot;,
       &quot;usageBytes&quot;: 0,
       &quot;workingSetBytes&quot;: 0,
       &quot;pageFaults&quot;: 0,
       &quot;majorPageFaults&quot;: 0
      },
      &quot;rootfs&quot;: {
       &quot;availableBytes&quot;: 18328821760,
       &quot;capacityBytes&quot;: 52710469632
      },
      &quot;logs&quot;: {
       &quot;availableBytes&quot;: 18328821760,
       &quot;capacityBytes&quot;: 52710469632
      },
      &quot;userDefinedMetrics&quot;: null
     }
    ],
    &quot;network&quot;: {
     &quot;time&quot;: &quot;2016-03-07T17:55:03Z&quot;,
     &quot;rxBytes&quot;: 90,
     &quot;rxErrors&quot;: 0,
     &quot;txBytes&quot;: 168,
     &quot;txErrors&quot;: 0
    }
   }
  ]
 }"><pre class="notranslate"><code class="notranslate">$ curl http://127.0.0.1:10255/stats/summary
{
  "node": {
   "nodeName": "127.0.0.1",
   "systemContainers": [
    {
     "name": "kubelet",
     "startTime": "2016-03-07T14:17:36Z",
     "cpu": {
      "time": "2016-03-07T17:55:11Z",
      "usageNanoCores": 221608525,
      "usageCoreNanoSeconds": 7174601461254
     },
     "memory": {
      "time": "2016-03-07T17:55:11Z",
      "usageBytes": 8132644864,
      "workingSetBytes": 5518856192,
      "pageFaults": 63072461,
      "majorPageFaults": 10940
     },
     "rootfs": {
      "availableBytes": 18328821760,
      "capacityBytes": 52710469632
     },
     "logs": {
      "availableBytes": 18328821760,
      "capacityBytes": 52710469632
     },
     "userDefinedMetrics": null
    },
    {
     "name": "runtime",
     "startTime": "2016-03-07T17:55:16Z",
     "cpu": {
      "time": "2016-03-07T17:55:00Z",
      "usageNanoCores": 17139366,
      "usageCoreNanoSeconds": 213037971066
     },
     "memory": {
      "time": "2016-03-07T17:55:00Z",
      "usageBytes": 31969280,
      "workingSetBytes": 25382912,
      "pageFaults": 9133069,
      "majorPageFaults": 14
     },
     "rootfs": {
      "availableBytes": 18328821760,
      "capacityBytes": 52710469632
     },
     "logs": {
      "availableBytes": 18328821760,
      "capacityBytes": 52710469632
     },
     "userDefinedMetrics": null
    }
   ],
   "startTime": "2016-03-07T14:17:36Z",
   "cpu": {
    "time": "2016-03-07T17:55:11Z",
    "usageNanoCores": 221608525,
    "usageCoreNanoSeconds": 7174601461254
   },
   "memory": {
    "time": "2016-03-07T17:55:11Z",
    "usageBytes": 8132644864,
    "workingSetBytes": 5518856192,
    "pageFaults": 63072461,
    "majorPageFaults": 10940
   },
   "fs": {
    "availableBytes": 18328821760,
    "capacityBytes": 52710469632,
    "usedBytes": 31680516096
   }
  },
  "pods": [
   {
    "podRef": {
     "name": "nginx-2040093540-7uo6d",
     "namespace": "default",
     "uid": "a514f7cf-e48d-11e5-bd2c-28d2444e470d"
    },
    "startTime": "2016-03-07T17:55:02Z",
    "containers": [
     {
      "name": "nginx",
      "startTime": "2016-03-07T17:55:04Z",
      "cpu": {
       "time": "2016-03-07T17:55:04Z",
       "usageCoreNanoSeconds": 21626484
      },
      "memory": {
       "time": "2016-03-07T17:55:04Z",
       "usageBytes": 8437760,
       "workingSetBytes": 2482176,
       "pageFaults": 883,
       "majorPageFaults": 48
      },
      "rootfs": {
       "availableBytes": 18328821760,
       "capacityBytes": 52710469632
      },
      "logs": {
       "availableBytes": 18328821760,
       "capacityBytes": 52710469632
      },
      "userDefinedMetrics": null
     },
     {
      "name": "nginx",
      "startTime": "2016-03-07T17:55:04Z",
      "cpu": {
       "time": "2016-03-07T17:55:04Z",
       "usageCoreNanoSeconds": 0
      },
      "memory": {
       "time": "2016-03-07T17:55:04Z",
       "usageBytes": 0,
       "workingSetBytes": 0,
       "pageFaults": 0,
       "majorPageFaults": 0
      },
      "rootfs": {
       "availableBytes": 18328821760,
       "capacityBytes": 52710469632
      },
      "logs": {
       "availableBytes": 18328821760,
       "capacityBytes": 52710469632
      },
      "userDefinedMetrics": null
     }
    ],
    "network": {
     "time": "2016-03-07T17:55:03Z",
     "rxBytes": 90,
     "rxErrors": 0,
     "txBytes": 168,
     "txErrors": 0
    }
   }
  ]
 }
</code></pre></div>
<p dir="auto">I debugged the issue down to here:<br>
<a href="https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/server/stats/summary.go#L202">https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/server/stats/summary.go#L202</a></p>
<p dir="auto">After adding more trace to this code block, I saw that nginx was being found twice for duplicate keys in the for loop.</p>
<p dir="auto">The first time:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="I0307 12:55:16.210107    1856 summary.go:189] STATS buildSummaryPods - key /system.slice/docker-37515a45e527fd66a5ab125f689d0113e2ded24bba6b25453d3d78f66fb1f932.scope
I0307 12:55:16.210115    1856 summary.go:196] STATS buildSummaryPods - ref {nginx-2040093540-7uo6d default a514f7cf-e48d-11e5-bd2c-28d2444e470d}"><pre class="notranslate"><code class="notranslate">I0307 12:55:16.210107    1856 summary.go:189] STATS buildSummaryPods - key /system.slice/docker-37515a45e527fd66a5ab125f689d0113e2ded24bba6b25453d3d78f66fb1f932.scope
I0307 12:55:16.210115    1856 summary.go:196] STATS buildSummaryPods - ref {nginx-2040093540-7uo6d default a514f7cf-e48d-11e5-bd2c-28d2444e470d}
</code></pre></div>
<p dir="auto">That makes sense because that is the cgroup for the container as reported by docker.</p>
<p dir="auto">The second time it appeared in the same for loop was for the key:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="I0307 12:55:16.210413    1856 summary.go:189] STATS buildSummaryPods - key /system.slice/var-lib-docker-devicemapper-mnt-37515a45e527fd66a5ab125f689d0113e2ded24bba6b25453d3d78f66fb1f932.mount
I0307 12:55:16.210416    1856 summary.go:196] STATS buildSummaryPods - ref {nginx-2040093540-7uo6d default a514f7cf-e48d-11e5-bd2c-28d2444e470d}
I0307 12:55:16.210420    1856 summary.go:205] STATS buildSummaryPods - podStats &amp;{{nginx-2040093540-7uo6d default a514f7cf-e48d-11e5-bd2c-28d2444e470d} 2016-03-07 17:55:02.669481834 +0000 UTC [{nginx 2016-03-07 17:55:04.121735851 +0000 UTC 0xc820e70030 0xc820826100 0xc8208ab7a0 0xc8208ab760 []}] 0xc8208261c0 []}
I0307 12:55:16.210429    1856 summary.go:209] STATS containerName nginx"><pre class="notranslate"><code class="notranslate">I0307 12:55:16.210413    1856 summary.go:189] STATS buildSummaryPods - key /system.slice/var-lib-docker-devicemapper-mnt-37515a45e527fd66a5ab125f689d0113e2ded24bba6b25453d3d78f66fb1f932.mount
I0307 12:55:16.210416    1856 summary.go:196] STATS buildSummaryPods - ref {nginx-2040093540-7uo6d default a514f7cf-e48d-11e5-bd2c-28d2444e470d}
I0307 12:55:16.210420    1856 summary.go:205] STATS buildSummaryPods - podStats &amp;{{nginx-2040093540-7uo6d default a514f7cf-e48d-11e5-bd2c-28d2444e470d} 2016-03-07 17:55:02.669481834 +0000 UTC [{nginx 2016-03-07 17:55:04.121735851 +0000 UTC 0xc820e70030 0xc820826100 0xc8208ab7a0 0xc8208ab760 []}] 0xc8208261c0 []}
I0307 12:55:16.210429    1856 summary.go:209] STATS containerName nginx
</code></pre></div>
<p dir="auto">It appears that we are treating the following as a different container in the pod summary:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/system.slice/var-lib-docker-devicemapper-mnt-37515a45e527fd66a5ab125f689d0113e2ded24bba6b25453d3d78f66fb1f932.mount"><pre class="notranslate"><code class="notranslate">/system.slice/var-lib-docker-devicemapper-mnt-37515a45e527fd66a5ab125f689d0113e2ded24bba6b25453d3d78f66fb1f932.mount
</code></pre></div>
<p dir="auto">Looking on the file system, I see:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cat /sys/fs/cgroup/cpu/system.slice/var-lib-docker-devicemapper-mnt-37515a45e527fd66a5ab125f689d0113e2ded24bba6b25453d3d78f66fb1f932.mount/"><pre class="notranslate"><code class="notranslate">cat /sys/fs/cgroup/cpu/system.slice/var-lib-docker-devicemapper-mnt-37515a45e527fd66a5ab125f689d0113e2ded24bba6b25453d3d78f66fb1f932.mount/
</code></pre></div>
<p dir="auto">Still investigating this some more, but looks like we are getting tripped up in cadvisor by this...</p>
<p dir="auto">/cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/vishh/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/vishh">@vishh</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/dchen1107/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dchen1107">@dchen1107</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ncdc/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ncdc">@ncdc</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sjenning/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sjenning">@sjenning</a></p>
<p dir="auto">Right now, we have the concept packing api query parameters into a struct, converting that to the versioned (serialized wire) format struct, then packing that into URL query parameters. This needs some slight adjustment for API groups.</p>
<p dir="auto">Options:</p>
<ol dir="auto">
<li>Change nothing; every group needs its own e.g. ListOptions struct in the internal and external versions.</li>
<li>Move common options (e.g. ListOptions) to unversioned.</li>
</ol>
<p dir="auto">If we do 2, then we need to modify the code here: <a href="https://github.com/kubernetes/kubernetes/blob/master/pkg/client/unversioned/request.go#L414">https://github.com/kubernetes/kubernetes/blob/master/pkg/client/unversioned/request.go#L414</a></p>
<p dir="auto">Options for modifying this code:</p>
<ol dir="auto">
<li>RESTClient takes a second "options" codec, which takes a struct, returns JSON, which can be converted to query parameters. (must be a second codec because the codec for the body of the request may be binary.)</li>
<li>RESTClient takes a second "options" codec, which takes a struct and returns URL query parameters directly.</li>
<li>Develop the concept of internal unversioned vs. external unversioned (UGH).</li>
</ol>
<p dir="auto">2.1 &amp; 2.2 both have the advantage that RESTClient doesn't need to know about "convertor" at all any more.</p>
<p dir="auto">2.1 can be done just by reusing our existing codec, which is nice, but 2.2 is probably clearer? 2.3 is confusing. Option 1 is verbose and error prone.</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/smarterclayton/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/smarterclayton">@smarterclayton</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/krousey/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/krousey">@krousey</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/wojtek-t/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/wojtek-t">@wojtek-t</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/caesarxuchao/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/caesarxuchao">@caesarxuchao</a> please all of you argue about this :)</p>