Answers: <p>Based on what I see so far, I have found the root cause, renamed kubelet service from old chart to new that can get targeted by serviceMonitors. So for me the best solution would be grafana kube-state-metrics + comparing what I see in the jvm dashboard</p>
  
Answers: CC BY-SA 4.0
Answers: 2020-10-15T15:17:52.810Z
Answers: 2020-10-15T15:17:52.810Z
Answers: anVzdGFub3RoZXJodW1hbg
Body: <p>micrometer exposing actuator metrics to set request/limit to pods in K8svs metrics-server vs kube-state-metrics -&gt; K8s Mixin from kube-promethteus-stack Grafana dashboad
  It's really blurry and frustrating to me to understand why there is such big difference between values from the 3 in the title and how should one utilize K8s Mixin to set proper request/limits and if that is expected at al.
  I was hoping I can just see same data that I see when I type kubectl top podname --containers to what I see when I open K8s -&gt; ComputeResources -&gt; Pods dashboard in Grafana. But not only the values differ by more than a double, but also reported values from actuator differ from both.
  When exposing spring data with micrometer the sum of jvm_memory_used_bytes is corresponding more to what I get from metrics-server (0.37.0) rather then what I see on Grafana from the mixin dashboards, but it is still far off.
  I am using K8s: 1.14.3 on Ubuntu 18.04 LTS managed by kubespray.
  kube-prometheus-stack 9.4.4 installed with helm 2.14.3.
  Spring boot 2.0 with Micrometer. I saw the explanation on metrics-server git that this is the value that kubelet use for OOMKill, but again this is not helpful at all as what should I do with the dashboard? What is the the way to handle this?</p>
  <p><a href="https://i.sstatic.net/aVzeS.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/aVzeS.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/elHG4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/elHG4.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/m1DZm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/m1DZm.png" alt="enter image description here" /></a></p>
  
Comments: CC BY-SA 4.0
Comments: 2020-10-15T14:42:26.897Z
Comments: I found out the double values are result of something in the provided querry in Grafana: When I run the following instant vector container_memory_working_set_bytes{cluster="", namespace="qa2", pod="service-x-684f5565b7-wbgm6", container!="POD", container!="", image!=""} why would I get 4 results instead of 2 (service and istio-proxy), what is the logical explanation behind this, I get 2 time the service and 2 time the proxy with same values, but different id in the metrics endpoint is that a bug in kube-prometheus-stack helm chart or perhaps something on operator level?
Comments: anVzdGFub3RoZXJodW1hbg
Tags: <kubernetes><prometheus><java><grafana>
Title: micrometer exposing actuator metrics vs kube-state-metrics vs metrics-server to set pod request/limits
UserDisplayName: anVzdGFub3RoZXJodW1hbg
ThreadText: # micrometer exposing actuator metrics vs kube-state-metrics vs metrics-server to set pod request/limits
  
  (Asked by: username_0 on 2020-10-15)
  
  micrometer exposing actuator metrics to set request/limit to pods in K8svs metrics-server vs kube-state-metrics -> K8s Mixin from kube-promethteus-stack Grafana dashboad
  It's really blurry and frustrating to me to understand why there is such big difference between values from the 3 in the title and how should one utilize K8s Mixin to set proper request/limits and if that is expected at al.
  I was hoping I can just see same data that I see when I type kubectl top podname --containers to what I see when I open K8s -> ComputeResources -> Pods dashboard in Grafana. But not only the values differ by more than a double, but also reported values from actuator differ from both.
  When exposing spring data with micrometer the sum of jvm\_memory\_used\_bytes is corresponding more to what I get from metrics-server (0.37.0) rather then what I see on Grafana from the mixin dashboards, but it is still far off.
  I am using K8s: 1.14.3 on Ubuntu 18.04 LTS managed by kubespray.
  kube-prometheus-stack 9.4.4 installed with helm 2.14.3.
  Spring boot 2.0 with Micrometer. I saw the explanation on metrics-server git that this is the value that kubelet use for OOMKill, but again this is not helpful at all as what should I do with the dashboard? What is the the way to handle this?
  
  [![enter image description here](https://i.sstatic.net/aVzeS.png)](https://i.sstatic.net/aVzeS.png)
  
  [![enter image description here](https://i.sstatic.net/elHG4.png)](https://i.sstatic.net/elHG4.png)
  
  [![enter image description here](https://i.sstatic.net/m1DZm.png)](https://i.sstatic.net/m1DZm.png)
  
  **Tags:** kubernetes, prometheus, java, grafana
  
  > **Comments:**
  > **username_0**: I found out the double values are result of something in the provided querry in Grafana: When I run the following instant vector container_memory_working_set_bytes{cluster="", namespace="qa2", pod="service-x-684f5565b7-wbgm6", container!="POD", container!="", image!=""} why would I get 4 results instead of 2 (service and istio-proxy), what is the logical explanation behind this, I get 2 time the service and 2 time the proxy with same values, but different id in the metrics endpoint is that a bug in kube-prometheus-stack helm chart or perhaps something on operator level?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-10-15. Score: 0
  
  Based on what I see so far, I have found the root cause, renamed kubelet service from old chart to new that can get targeted by serviceMonitors. So for me the best solution would be grafana kube-state-metrics + comparing what I see in the jvm dashboard
  
  
  ---
  
  