Answers: <p>Multiple things here:</p>
  
  <ol>
  <li><p>Youre on AWS ec2, hence whatever you are doing on your instance for measuring CPU is calculating CPU on hypervisor level and not instance level. To very this run any load test and check iostat -ct 1 and CPU usage in cloudwatch. CPU usage in cloudwatch is always 10-20% greater than what iostat will report and that is because iostat will give CPU usage on hypervisor level.</p></li>
  <li><p>As docker to see how kubernetes and docker metrics compare i suggest to run the containers with --cpuset=1 or any number to allow all containers to use only a single vCPU.</p></li>
  <li><p>Also in AWS 1 CPU= 2vcpu. It is hyperthreaded to 2. You can maybe take this into consideration while calculating.</p></li>
  <li><p>Finally the best metric to use to see CPU usage for a particular application is using htop and correlating with your cloudwatch metrics.</p></li>
  <li><p>I have also observed sometimes docker daemon pinning itself to one of the virtual CPUs and hence when you are reducing it to 1000m, maybe the whole setup is getting slow because the reduction is happening on any one of the vpcus. You can use mpstat to get into details of this. </p></li>
  <li><p>Lastly on the host level you can pin docker to a single CPU and observe more.</p></li>
  </ol>
  
  <p>Hope this brings you somewhat closer. Update me if you have already found a solution.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-07-28T12:11:56.537Z
Answers: 2017-07-28T12:11:56.537Z
Answers: lakshayk
Body: <p>We recently switched over our production environment to Kubernetes. I'd like to enforce CPU limits on the containers. I'm getting conflicting CPU metrics that do not fit together. Here's my setup:</p>
  
  <ul>
  <li>DataDog agents running as a <code>Daemonset</code></li>
  <li>Existing applications running without CPU Limits</li>
  <li>Containers in question are multi-threaded Ruby applications</li>
  <li>Two metrics: <code>kubernetes.cpu.usage.{avg,max}</code> and <code>docker.cpu.usage</code></li>
  <li><code>c4.xlarge</code> cluster nodes (4 vCPUs or 4000m in Kubernetes terms)</li>
  </ul>
  
  <p><code>kubernetes.cpu.usage.max</code> reports ~600m for the containers in question. <code>docker.cpu.usage</code> reports ~60%. It follows that a 1000m CPU limit would be more than enough capacity under normal operation. </p>
  
  <p>I set the limit to 1000m. Then <code>docker.container.throttles</code> goes up significantly while <code>kubernetes.cpu.usage.max</code> and <code>docker.cpu.usage</code> stay the same. The system all fells to it's knees during this time. This does not make sense to me.</p>
  
  <p>I researched Docker stats. It seems that <code>docker stats</code> (and the underlying API) <em>normalize</em> load according to CPU cores. So in my case, <code>docker.cpu.usage</code> of 60% comes (4000m * 0.60) to 2400m in Kubernetes terms. However this does not correlate with any Kubernetes numbers. I did another experiment to test my hypothesis that Kubernetes numbers are incorrect. I set the limit to 2600m (for some extra headroom). This did not result in any throttles. However Kubernetes observed CPU usage did not change. This leaves me confused.</p>
  
  <p>So my questions are:</p>
  
  <ul>
  <li>Does this feel like a bug in Kubernetes (or something in the stack?)</li>
  <li>Is my understanding correct?</li>
  </ul>
  
  <p>My follow up question relates to how to properly determine CPU for Ruby applications. One container uses Puma. This is a multi-threaded web server with a configurable amount of threads. HTTP requests are handled by one of the threads. The second application is a thrift server using the threaded server. Each incoming TCP connection is handled by it's own thread. The thread exits when the connection closes. Ruby as GIL (Global Interpreter Lock) so only one thread can execute Ruby code at a time. That does allow for multiple threads executing IO and things like that.</p>
  
  <p>I think the best approach is limit the number of threads running in each application and approximating Kubernetes CPU limits based on the number of threads. The processes are not forking so total CPU usage is harder to predict. </p>
  
  <p>The question here is: how to properly predict CPU usage and limits for these applications?</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-07-03T15:01:32.090Z
Comments: Did you try on a 1cpu node and a 2 cpu node to see how the number correlates (or not) ?
Comments: Tensibai
Tags: <docker><kubernetes>
Title: Conflicting Kubernetes CPU usage & Docker Container Metrics
UserDisplayName: ahawkins
ThreadText: # Conflicting Kubernetes CPU usage & Docker Container Metrics
  
  (Asked by: username_0 on 2017-07-01)
  
  We recently switched over our production environment to Kubernetes. I'd like to enforce CPU limits on the containers. I'm getting conflicting CPU metrics that do not fit together. Here's my setup:
  
  * DataDog agents running as a `Daemonset`
  * Existing applications running without CPU Limits
  * Containers in question are multi-threaded Ruby applications
  * Two metrics: `kubernetes.cpu.usage.{avg,max}` and `docker.cpu.usage`
  * `c4.xlarge` cluster nodes (4 vCPUs or 4000m in Kubernetes terms)
  
  `kubernetes.cpu.usage.max` reports ~600m for the containers in question. `docker.cpu.usage` reports ~60%. It follows that a 1000m CPU limit would be more than enough capacity under normal operation.
  
  I set the limit to 1000m. Then `docker.container.throttles` goes up significantly while `kubernetes.cpu.usage.max` and `docker.cpu.usage` stay the same. The system all fells to it's knees during this time. This does not make sense to me.
  
  I researched Docker stats. It seems that `docker stats` (and the underlying API) *normalize* load according to CPU cores. So in my case, `docker.cpu.usage` of 60% comes (4000m \* 0.60) to 2400m in Kubernetes terms. However this does not correlate with any Kubernetes numbers. I did another experiment to test my hypothesis that Kubernetes numbers are incorrect. I set the limit to 2600m (for some extra headroom). This did not result in any throttles. However Kubernetes observed CPU usage did not change. This leaves me confused.
  
  So my questions are:
  
  * Does this feel like a bug in Kubernetes (or something in the stack?)
  * Is my understanding correct?
  
  My follow up question relates to how to properly determine CPU for Ruby applications. One container uses Puma. This is a multi-threaded web server with a configurable amount of threads. HTTP requests are handled by one of the threads. The second application is a thrift server using the threaded server. Each incoming TCP connection is handled by it's own thread. The thread exits when the connection closes. Ruby as GIL (Global Interpreter Lock) so only one thread can execute Ruby code at a time. That does allow for multiple threads executing IO and things like that.
  
  I think the best approach is limit the number of threads running in each application and approximating Kubernetes CPU limits based on the number of threads. The processes are not forking so total CPU usage is harder to predict.
  
  The question here is: how to properly predict CPU usage and limits for these applications?
  
  **Tags:** docker, kubernetes
  
  > **Comments:**
  > **username_1**: Did you try on a 1cpu node and a 2 cpu node to see how the number correlates (or not) ?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2017-07-28. Score: 4
  
  Multiple things here:
  
  1. Youre on AWS ec2, hence whatever you are doing on your instance for measuring CPU is calculating CPU on hypervisor level and not instance level. To very this run any load test and check iostat -ct 1 and CPU usage in cloudwatch. CPU usage in cloudwatch is always 10-20% greater than what iostat will report and that is because iostat will give CPU usage on hypervisor level.
  2. As docker to see how kubernetes and docker metrics compare i suggest to run the containers with --cpuset=1 or any number to allow all containers to use only a single vCPU.
  3. Also in AWS 1 CPU= 2vcpu. It is hyperthreaded to 2. You can maybe take this into consideration while calculating.
  4. Finally the best metric to use to see CPU usage for a particular application is using htop and correlating with your cloudwatch metrics.
  5. I have also observed sometimes docker daemon pinning itself to one of the virtual CPUs and hence when you are reducing it to 1000m, maybe the whole setup is getting slow because the reduction is happening on any one of the vpcus. You can use mpstat to get into details of this.
  6. Lastly on the host level you can pin docker to a single CPU and observe more.
  
  Hope this brings you somewhat closer. Update me if you have already found a solution.
  
  
  ---
  
  