Answers: <p>I found out what was happening in my graphs. All the Pods mentioned above have one thing in common: they use the host's network namespace, so their network metrics are <em>all identical</em>, and equal to the global <em>host's metric</em> (just with a slightly different precision).</p>
  <pre><code>$ kubectl - monitoring get pod -o jsonpath='{.spec.hostNetwork}' \
      prometheus-stack-prometheus-node-exporter-jnhw7
  
  true
  </code></pre>
  <pre><code>$ kubectl -n kube-system get pod -o jsonpath='{.items[*].spec.hostNetwork}' \
      kube-proxy-gke-triggermesh-product-control-plane-7fc0ad24-z586 \
      gke-metrics-agent-5cv4m \
      prometheus-to-sd-tk8jv \
      fluentbit-gke-xh879
  
  true true true true 
  </code></pre>
  <p>One way to see it is to compare the host's metric to <em>one</em> of the above Pods:</p>
  <p><a href="https://i.sstatic.net/RHQiT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/RHQiT.png" alt="Comparison of eth0 and a Pod with hostNetwork" /></a></p>
  
Answers: CC BY-SA 4.0
Answers: 2022-11-11T13:50:21.313Z
Answers: Thanks for the hint. We researched a similar problem today. Our node-exporter pod bandwith massively increased (more than 400mb/s) - which we could not explain. In our case a redis pod had very high network transmit, which was routed trough calico/kube-proxy in the hostNetwork.
  
  Although the host machine had only 150mb/s network transmit, which was caused by calico which only routed external traffic to other nodes. Lots of traffic stayed inside the virtual network of the host itself.
Answers: Martin Lantzsch
Answers: CC BY-SA 4.0
Answers: 2020-11-20T19:43:10.713Z
Answers: 2020-11-20T19:43:10.713Z
Answers: Antoine Cotten
Body: <h3>Context</h3>
  <p>I am trying to chart the network bandwidth usage of a node in 2 different manners:</p>
  <ol>
  <li>By looking at global metrics for that node</li>
  <li>By summing up the corresponding metric for each Pod</li>
  </ol>
  <p>To achieve this, I am issuing the following Prometheus queries (<em>example for the receive bandwidth</em>):</p>
  <ul>
  <li><p>For the entire node (<em>metric from node-exporter</em>)</p>
  <pre><code>sum(irate(node_network_receive_bytes_total{instance=&quot;10.142.0.54:9100&quot;}[$__rate_interval])) by (device)
  </code></pre>
  </li>
  <li><p>Per Pod (<em>metric from kubelet</em>)</p>
  <pre><code>sum(irate(container_network_receive_bytes_total{node=&quot;$node&quot;,container!=&quot;&quot;}[$__rate_interval])) by (pod,interface)
  </code></pre>
  </li>
  </ul>
  <p>The results are displayed in the following Grafana dashboard, after generating some load on a HTTP service called <code>thrpt-receiver</code>:</p>
  <p><a href="https://i.sstatic.net/sZtI5.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/sZtI5.png" alt="Receive bandwidth" /></a></p>
  <p>Here's what I see if I look at the raw metrics, without <code>sum()</code> and <code>irate()</code> applied:</p>
  <p><a href="https://i.sstatic.net/ASC5X.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ASC5X.png" alt="Received bytes" /></a></p>
  <h3>Problem</h3>
  <p>As you can see, results are vastly different, to the point I'm almost certain I am doing something wrong, but what?</p>
  <p>What makes me especially suspicious about the Pod metrics is the supposedly increasing received bandwidth of kube-proxy (which AFAIK is not supposed to be receiving any traffic in iptables mode), and agents such as the Prometheus node-exporter, etc.</p>
  
Tags: <prometheus>
Title: Node-centric network metrics are very different from the sum of metrics of the Pods running on that node
UserDisplayName: Antoine Cotten
ThreadText: # Node-centric network metrics are very different from the sum of metrics of the Pods running on that node
  
  (Asked by: username_0 on 2020-11-20)
  
  ### Context
  
  I am trying to chart the network bandwidth usage of a node in 2 different manners:
  
  1. By looking at global metrics for that node
  2. By summing up the corresponding metric for each Pod
  
  To achieve this, I am issuing the following Prometheus queries (*example for the receive bandwidth*):
  
  * For the entire node (*metric from node-exporter*)
  
    ```
    sum(irate(node_network_receive_bytes_total{instance="10.142.0.54:9100"}[$__rate_interval])) by (device)
  
    ```
  * Per Pod (*metric from kubelet*)
  
    ```
    sum(irate(container_network_receive_bytes_total{node="$node",container!=""}[$__rate_interval])) by (pod,interface)
  
    ```
  
  The results are displayed in the following Grafana dashboard, after generating some load on a HTTP service called `thrpt-receiver`:
  
  [![Receive bandwidth](https://i.sstatic.net/sZtI5.png)](https://i.sstatic.net/sZtI5.png)
  
  Here's what I see if I look at the raw metrics, without `sum()` and `irate()` applied:
  
  [![Received bytes](https://i.sstatic.net/ASC5X.png)](https://i.sstatic.net/ASC5X.png)
  
  ### Problem
  
  As you can see, results are vastly different, to the point I'm almost certain I am doing something wrong, but what?
  
  What makes me especially suspicious about the Pod metrics is the supposedly increasing received bandwidth of kube-proxy (which AFAIK is not supposed to be receiving any traffic in iptables mode), and agents such as the Prometheus node-exporter, etc.
  
  **Tags:** prometheus
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-11-20. Score: 1 | Accepted answer
  
  I found out what was happening in my graphs. All the Pods mentioned above have one thing in common: they use the host's network namespace, so their network metrics are *all identical*, and equal to the global *host's metric* (just with a slightly different precision).
  
  ```
  $ kubectl - monitoring get pod -o jsonpath='{.spec.hostNetwork}' \
      prometheus-stack-prometheus-node-exporter-jnhw7
  
  true
  
  ```
  
  ```
  $ kubectl -n kube-system get pod -o jsonpath='{.items[*].spec.hostNetwork}' \
      kube-proxy-gke-triggermesh-product-control-plane-7fc0ad24-z586 \
      gke-metrics-agent-5cv4m \
      prometheus-to-sd-tk8jv \
      fluentbit-gke-xh879
  
  true true true true 
  
  ```
  
  One way to see it is to compare the host's metric to *one* of the above Pods:
  
  [![Comparison of eth0 and a Pod with hostNetwork](https://i.sstatic.net/RHQiT.png)](https://i.sstatic.net/RHQiT.png)
  
  > **Comments:**
  > **username_1**: Thanks for the hint. We researched a similar problem today. Our node-exporter pod bandwith massively increased (more than 400mb/s) - which we could not explain. In our case a redis pod had very high network transmit, which was routed trough calico/kube-proxy in the hostNetwork.  Although the host machine had only 150mb/s network transmit, which was caused by calico which only routed external traffic to other nodes. Lots of traffic stayed inside the virtual network of the host itself.  
  
  ---
  
  