Answers: <p>To piggy back off of @eduardo-baitello answer, A third option is to increase the <code>CPUThrottlingPercent</code> config <a href="https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/f7ca48cca5d9cadc9a2203b8c0b3bb3eb85f3294/config.libsonnet#L44" rel="nofollow noreferrer">here</a></p>
  
Answers: CC BY-SA 4.0
Answers: 2021-02-08T23:28:40.710Z
Answers: Mind posting this as a comment?
Answers: codeforester
Answers: CC BY-SA 4.0
Answers: 2021-02-08T19:12:58.913Z
Answers: 2021-02-08T19:12:58.913Z
Answers: Saleh Rastani
Answers: <p>The <code>CPUThrottlingHigh</code> is an alert created by the <a href="https://github.com/kubernetes-monitoring/kubernetes-mixin" rel="noreferrer">kubernetes-mixin</a> project. There is an open issue (<a href="https://github.com/kubernetes-monitoring/kubernetes-mixin/issues/108" rel="noreferrer">#108</a>) to discuss this alert. I suggest that you read all the comments on this issue to better understand the problem.</p>
  <p>In short, the problem is: When working with low CPU limits, spiky workloads can have low averages and still be being throttled.</p>
  <p>Also, take a look at this issue (<a href="https://github.com/kubernetes/kubernetes/issues/67577" rel="noreferrer">#67577</a>) from Kubernetes project, which addresses a Kernel bug in CFS quotas that may cause unnecessary CPU throttling. The discussion is still open, and the Kubernetes project are even considering disabling CFS quotas for pods in the <a href="https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#create-a-pod-that-gets-assigned-a-qos-class-of-guaranteed" rel="noreferrer"><code>Guaranteed</code> QoS</a> (see <a href="https://github.com/kubernetes/kubernetes/issues/70585" rel="noreferrer">#70585</a> for reference).</p>
  <p>Consider the following options:</p>
  <ul>
  <li>Increase (or even remove) your container CPU limits</li>
  <li>Disable Kubernetes CFS quotas entirely
  (kubelet's flag <code>--cpu-cfs-quota=false</code>)</li>
  <li>Use a Kernel version that contains this fix (<a href="https://github.com/torvalds/linux/commit/512ac999d2755d2b7109e996a76b6fb8b888631d" rel="noreferrer">torvalds/linux 512ac99</a>)</li>
  </ul>
  
Answers: CC BY-SA 4.0
Answers: 2019-04-03T14:18:15.360Z
Answers: Thanks, it is nice to know that the issue is not on my installation. I don't want to increase or remove limits for now. And I am using AKS (a managed implementation of Kubernetes in Azure cloud) so I cannot disable CFS quotas. As per the comments in the issue, I am going to try to update the kernel version to see what happens.
Answers: jobou
Answers: CC BY-SA 4.0
Answers: 2019-12-27T10:16:43.393Z
Answers: @jobou, have you update a kernel version? Was it helpful?
Answers: Vadim Yangunaev
Answers: CC BY-SA 4.0
Answers: 2019-12-28T19:16:11.140Z
Answers: No sorry, I have not worked on this issue yet so I don't have any real feedback to give.
Answers: jobou
Answers: CC BY-SA 4.0
Answers: 2020-12-14T21:33:42.603Z
Answers: We are still eeing this exact same thing in Dec 2020 and we are running on GKE 1.16.15.4300.  Our requests and limits are quite a bit higher than the ones in the question.  Removing CPU limits is not really an option for us.
Answers: Jon Carlson
Answers: CC BY-SA 4.0
Answers: 2019-04-01T23:10:18.107Z
Answers: 2021-05-26T15:46:59.860Z
Answers: 2021-05-26T15:46:59.860Z
Answers: Eduardo Baitello
Body: <p>I have installed Prometheus to monitor my installation and it is frequently raising alerts about CPU throttling.</p>
  
  <p>The Prometheus alert rules to identify this alert is :</p>
  
  <pre><code>alert: CPUThrottlingHigh
  expr: 100
    * sum by(container_name, pod_name, namespace) (increase(container_cpu_cfs_throttled_periods_total{container_name!=""}[5m]))
    / sum by(container_name, pod_name, namespace) (increase(container_cpu_cfs_periods_total[5m]))
    &gt; 25
  for: 15m
  </code></pre>
  
  <p>If I look at one of the pods identified by this alert, it does not seem to have any reason to throttle :</p>
  
  <pre><code>$ kubectl top pod -n monitoring my-pod
  NAME            CPU(cores)   MEMORY(bytes)   
  my-pod          0m           6Mi
  </code></pre>
  
  <p>This pod has one container with these resources setup :</p>
  
  <pre><code>Limits:
    cpu:     100m
    memory:  128Mi
  Requests:
    cpu:     25m
    memory:  64Mi
  </code></pre>
  
  <p>And the node that is hosting this pod is not under any heavy cpu use :</p>
  
  <pre><code>$ kubectl -n monitoring top node aks-agentpool-node-1
  NAME                       CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%   
  aks-agentpool-node-1       853m         21%       11668Mi         101%
  </code></pre>
  
  <p>On grafana, if I look at the chart for this pod, it never goes above <code>0,000022</code> of cpu usage</p>
  
  <p>Why is it throttling ?</p>
  
Comments: CC BY-SA 4.0
Comments: 2019-03-14T08:06:09.957Z
Comments: Have you validated that you need to multiple by 100? Is it a decimal format for the percentage or an integer value?
Comments: sheldonhull
Comments: CC BY-SA 4.0
Comments: 2019-03-14T11:15:16.427Z
Comments: I am not sure to understand. The expression `100m` is the same as `0.1` cpu.
Comments: jobou
Comments: CC BY-SA 4.0
Comments: 2019-03-14T13:51:14.833Z
Comments: If that's the syntax for Prometheus then disregard. Was just checking.
Comments: sheldonhull
Tags: <docker><kubernetes><monitoring><prometheus>
Title: Prometheus alert CPUThrottlingHigh raised but monitoring does not show it
UserDisplayName: jobou
ThreadText: # Prometheus alert CPUThrottlingHigh raised but monitoring does not show it
  
  (Asked by: username_0 on 2019-02-27)
  
  I have installed Prometheus to monitor my installation and it is frequently raising alerts about CPU throttling.
  
  The Prometheus alert rules to identify this alert is :
  
  ```
  alert: CPUThrottlingHigh
  expr: 100
    * sum by(container_name, pod_name, namespace) (increase(container_cpu_cfs_throttled_periods_total{container_name!=""}[5m]))
    / sum by(container_name, pod_name, namespace) (increase(container_cpu_cfs_periods_total[5m]))
    > 25
  for: 15m
  
  ```
  
  If I look at one of the pods identified by this alert, it does not seem to have any reason to throttle :
  
  ```
  $ kubectl top pod -n monitoring my-pod
  NAME            CPU(cores)   MEMORY(bytes)   
  my-pod          0m           6Mi
  
  ```
  
  This pod has one container with these resources setup :
  
  ```
  Limits:
    cpu:     100m
    memory:  128Mi
  Requests:
    cpu:     25m
    memory:  64Mi
  
  ```
  
  And the node that is hosting this pod is not under any heavy cpu use :
  
  ```
  $ kubectl -n monitoring top node aks-agentpool-node-1
  NAME                       CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%   
  aks-agentpool-node-1       853m         21%       11668Mi         101%
  
  ```
  
  On grafana, if I look at the chart for this pod, it never goes above `0,000022` of cpu usage
  
  Why is it throttling ?
  
  **Tags:** docker, kubernetes, monitoring, prometheus
  
  > **Comments:**
  > **username_1**: Have you validated that you need to multiple by 100? Is it a decimal format for the percentage or an integer value?  
  > **username_0**: I am not sure to understand. The expression `100m` is the same as `0.1` cpu.  
  > **username_1**: If that's the syntax for Prometheus then disregard. Was just checking.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2019-04-01. Score: 15 | Accepted answer
  
  The `CPUThrottlingHigh` is an alert created by the [kubernetes-mixin](https://github.com/kubernetes-monitoring/kubernetes-mixin) project. There is an open issue ([#108](https://github.com/kubernetes-monitoring/kubernetes-mixin/issues/108)) to discuss this alert. I suggest that you read all the comments on this issue to better understand the problem.
  
  In short, the problem is: When working with low CPU limits, spiky workloads can have low averages and still be being throttled.
  
  Also, take a look at this issue ([#67577](https://github.com/kubernetes/kubernetes/issues/67577)) from Kubernetes project, which addresses a Kernel bug in CFS quotas that may cause unnecessary CPU throttling. The discussion is still open, and the Kubernetes project are even considering disabling CFS quotas for pods in the [`Guaranteed` QoS](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#create-a-pod-that-gets-assigned-a-qos-class-of-guaranteed) (see [#70585](https://github.com/kubernetes/kubernetes/issues/70585) for reference).
  
  Consider the following options:
  
  * Increase (or even remove) your container CPU limits
  * Disable Kubernetes CFS quotas entirely
    (kubelet's flag `--cpu-cfs-quota=false`)
  * Use a Kernel version that contains this fix ([torvalds/linux 512ac99](https://github.com/torvalds/linux/commit/512ac999d2755d2b7109e996a76b6fb8b888631d))
  
  > **Comments:**
  > **username_0**: Thanks, it is nice to know that the issue is not on my installation. I don't want to increase or remove limits for now. And I am using AKS (a managed implementation of Kubernetes in Azure cloud) so I cannot disable CFS quotas. As per the comments in the issue, I am going to try to update the kernel version to see what happens.  
  > **username_5**: @username_0, have you update a kernel version? Was it helpful?  
  > **username_0**: No sorry, I have not worked on this issue yet so I don't have any real feedback to give.  
  > **username_6**: We are still eeing this exact same thing in Dec 2020 and we are running on GKE 1.16.15.4300.  Our requests and limits are quite a bit higher than the ones in the question.  Removing CPU limits is not really an option for us.  
  
  ---
  
  ### Answer by username_2 on 2021-02-08. Score: 2
  
  To piggy back off of @username_4-username_4 answer, A third option is to increase the `CPUThrottlingPercent` config [here](https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/f7ca48cca5d9cadc9a2203b8c0b3bb3eb85f3294/config.libsonnet#L44)
  
  > **Comments:**
  > **username_3**: Mind posting this as a comment?  
  
  ---
  
  