title: Kubernetes CPU throttling with CPU usage well below requests/limits
body: <p>I have setup CPU and Memory Requests=Limits on all containers of my pod in order to qualify it for Guaranteed Quality of Service class. Now, look at these CPU Usage and CPU Throttling graphs for same Pod for last 6 hours.</p>
  <p><a href="https://i.stack.imgur.com/EjGa7.png" rel="noreferrer"><img src="https://i.stack.imgur.com/EjGa7.png" alt="CPU Utilization and Throttling for a pod over 6 hrs period" /></a></p>
  <p>Does this look normal and expected?</p>
  <p>CPU Usage has not even touched 50% of the set limit a single time and still it was being throttled upto 58% at times.</p>
  <p>And a side question, what does that red line at 25% in the Throttling graph indicates?</p>
  <p>I did some research on this topic and found that there was a bug in Linux kernel that could have caused this and that it was fixed in version 4.18 of the kernel. Reference: <a href="https://github.com/kubernetes/kubernetes/issues/67577#issuecomment-466609030" rel="noreferrer">this</a> and <a href="https://github.com/kubernetes/kubernetes/issues/51135" rel="noreferrer">this</a></p>
  <p>We are on GKE running Container Optimized OS by Google. I checked the linux kernel version on our nodes and they are on 4.19.112+ so I guess we already have that patch? What else could be the reason of this throttling pattern?</p>
  <p>P.S. This pod (actually a deployment with autoscaling) is deployed on a separate Node pool which has none of our other workloads running on it. So the only pods other than this deployment running on Nodes in this node pool are some metrics and logging agents and exporters. Here is the full list of pods running on the same Node at which the pod discussed above is scheduled. There are indeed some pods that don't have any CPU limits set on them. Do I need to somehow set CPU limits on these as well?</p>
  <p><a href="https://i.stack.imgur.com/Q3GOF.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Q3GOF.png" alt="Pods running on a Node" /></a></p>
  <p>Our GKE version is <strong>1.16.9-gke.2</strong></p>
  <p>Here is the manifest file containing deployment, service, and auto scaler definitions.</p>
  <pre><code>apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: endpoints
    labels:
      app: endpoints
  spec:
    replicas: 2
    selector:
      matchLabels:
        run: endpoints
    strategy:
      rollingUpdate:
        maxSurge: 2
        maxUnavailable: 0
    template:
      metadata:
        labels:
          run: endpoints
      spec:
        terminationGracePeriodSeconds: 60
        containers:
          - name: endpoints
            image: gcr.io/&lt;PROJECT_ID&gt;/endpoints:&lt;RELEASE_VERSION_PLACEHOLDER&gt;
            livenessProbe:
              httpGet:
                path: /probes/live
                port: 8080
              initialDelaySeconds: 20
              timeoutSeconds: 5
            readinessProbe:
              httpGet:
                path: /probes/ready
                port: 8080
              initialDelaySeconds: 20
              timeoutSeconds: 5
            ports:
              - containerPort: 8080
                protocol: TCP
            env:
              - name: GOOGLE_APPLICATION_CREDENTIALS
                value: &quot;/path/to/secret/gke-endpoints-deployments-access.json&quot;
              - name: POD_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.name
              - name: POD_NAMESPACE_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.namespace
              - name: DEPLOYMENT_NAME
                value: &quot;endpoints&quot;
            resources:
              requests:
                memory: &quot;5Gi&quot;
                cpu: 2
              limits:
                memory: &quot;5Gi&quot;
                cpu: 2
            volumeMounts:
              - name: endpoints-gcp-access
                mountPath: /path/to/secret
                readOnly: true
            lifecycle:
              preStop:
                exec:
                  # SIGTERM triggers a quick exit; gracefully terminate instead
                  command: [&quot;/bin/sh&quot;,&quot;-c&quot;,&quot;sleep 3; /usr/sbin/nginx -s quit; sleep 57&quot;]
          # [START proxy_container]
          - name: cloudsql-proxy
            image: gcr.io/cloudsql-docker/gce-proxy:1.16
            command: [&quot;/cloud_sql_proxy&quot;,
                      &quot;-instances=&lt;PROJECT_ID&gt;:&lt;ZONE&gt;:prod-db=tcp:3306,&lt;PROJECT_ID&gt;:&lt;ZONE&gt;:prod-db-read-replica=tcp:3307&quot;,
                      &quot;-credential_file=/path/to/secret/gke-endpoints-deployments-access.json&quot;]
            # [START cloudsql_security_context]
            securityContext:
              runAsUser: 2  # non-root user
              allowPrivilegeEscalation: false
            # [END cloudsql_security_context]
            resources:
              requests:
                memory: &quot;50Mi&quot;
                cpu: 0.1
              limits:
                memory: &quot;50Mi&quot;
                cpu: 0.1
            volumeMounts:
              - name: endpoints-gcp-access
                mountPath: /path/to/secret
                readOnly: true
          # [END proxy_container]
          # [START nginx-prometheus-exporter container]
          - name: nginx-prometheus-exporter
            image: nginx/nginx-prometheus-exporter:0.7.0
            ports:
              - containerPort: 9113
                protocol: TCP
            env:
              - name: CONST_LABELS
                value: &quot;app=endpoints&quot;
            resources:
              requests:
                memory: &quot;50Mi&quot;
                cpu: 0.1
              limits:
                memory: &quot;50Mi&quot;
                cpu: 0.1
          # [END nginx-prometheus-exporter container]
        tolerations:
          - key: &quot;qosclass&quot;
            operator: &quot;Equal&quot;
            value: &quot;guaranteed&quot;
            effect: &quot;NoSchedule&quot;
        nodeSelector:
          qosclass: guaranteed
        # [START volumes]
        volumes:
          - name: endpoints-gcp-access
            secret:
              secretName: endpoints-gcp-access
        # [END volumes]
  ---
  apiVersion: cloud.google.com/v1beta1
  kind: BackendConfig
  metadata:
    name: endpoints-backendconfig
  spec:
    timeoutSec: 60
    connectionDraining:
      drainingTimeoutSec: 60
  ---
  apiVersion: v1
  kind: Service
  metadata:
    name: endpoints
    labels:
      app: endpoints
    annotations:
      cloud.google.com/neg: '{&quot;ingress&quot;: true}' # Creates a NEG after an Ingress is created
      beta.cloud.google.com/backend-config: '{&quot;ports&quot;: {&quot;80&quot;:&quot;endpoints-backendconfig&quot;}}'
  spec:
    type: NodePort
    selector:
      run: endpoints
    ports:
      - name: endpoints-nginx
        port: 80
        protocol: TCP
        targetPort: 8080
      - name: endpoints-metrics
        port: 81
        protocol: TCP
        targetPort: 9113
  ---
  apiVersion: autoscaling/v2beta1
  kind: HorizontalPodAutoscaler
  metadata:
    name: endpoints-autoscaler
  spec:
    minReplicas: 2
    maxReplicas: 10
    metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 40
    - type: External
      external:
        metricName: external.googleapis.com|prometheus|nginx_http_requests_total
        metricSelector:
          matchLabels:
            metric.labels.app: endpoints
        targetAverageValue: &quot;5&quot;
    scaleTargetRef:
      apiVersion: apps/v1
      kind: Deployment
      name: endpoints
  ---
  apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    name: endpoints-nginx-monitor
    namespace: monitoring
    labels:
      app: endpoints-nginx-monitor
      chart: prometheus-operator-8.13.7
      release: prom-operator
      heritage: Tiller
  spec:
    selector:
      matchLabels:
        app: endpoints
    namespaceSelector:
      any: true
    endpoints:
    - port: endpoints-metrics
      path: &quot;/metrics&quot;
  </code></pre>
  <p>And here is the dockerfile for the only custom container image used in the deployment:</p>
  <pre><code># Dockerfile extending the generic PHP image with application files for a
  # single application.
  FROM gcr.io/google-appengine/php:latest
  
  # The Docker image will configure the document root according to this
  # environment variable.
  ENV DOCUMENT_ROOT /app
  
  RUN /bin/bash /stackdriver-files/enable_stackdriver_integration.sh
  </code></pre>