Answers: <p>What does 'kubectl top node' return for your node usage?</p>
  <p>Kubernetes is pretty low overhead and can certainly handle more than 15req/s. Does your application rely on a database? If so, where is that database living? Is that single node also running any other pods?</p>
  <p>As others noted, an APM would be a helpful tool to diagnose unexpected performance issues.</p>
  
Answers: CC BY-SA 4.0
Answers: 2021-07-15T18:41:43.023Z
Answers: 2021-07-15T18:41:43.023Z
Answers: Jon W
Body: <p>I have a simple Django web application running on AWS EC2 instance (2 vCPU, 4Gi).
  For that type of deployment I have a performance (measured with <a href="https://github.com/rakyll/hey" rel="nofollow noreferrer">Hey</a>)</p>
  <pre><code>$ hey -t 30 -z 1m https://mydomain.com/my/endpoint/
  </code></pre>
  <p>about <code>30 reqs/s</code></p>
  <p>When deployed with Kubernetes onto EKS cluster (same docker image) I have about <code>15 reqs/s</code> for 1 pod with limits</p>
  <pre><code>    resources:
        requests:
          cpu: &quot;500m&quot;
          memory: &quot;2Gi&quot;
        limits:
          cpu: &quot;1&quot;
          memory: &quot;4Gi&quot;
  </code></pre>
  <p>I expect that scaling pods would double the performance but nothing happens (request rate stays the same)</p>
  <pre><code>$ kubectl scale deploy/myapp --replicas=2
  </code></pre>
  <p>I'm running Django app behind <code>gunicorn</code> like this (adjusting number of workers did not help)</p>
  <pre><code>gunicorn --config gunicorn.py myapp.wsgi:application
  # gunicorn.py
  import os
  
  # tried different values from 1 to 5
  # for 1 vCPU allocated for the pod it should be 3 as per docs
  # (2 x $num_cores) + 1
  # https://docs.gunicorn.org/en/stable/design.html#how-many-workers
  workers = int(os.getenv('GUNICORN_WORKERS_APP', default=4))
  daemon = False
  bind = '0.0.0.0:8080'
  max_requests = 2000
  </code></pre>
  <p>So couple of questions here:</p>
  <ul>
  <li>how to properly debug this type of issue and what's the toolset I can use?</li>
  <li>how <code>service</code> does load balancing to pods and is there any overhead introduced?</li>
  <li>is there an overhead for resource management (cpu allocation/throttling)?</li>
  <li>how performance tuning is done for k8s-driven applications?</li>
  </ul>
  
Comments: CC BY-SA 4.0
Comments: 2021-07-07T20:56:59.697Z
Comments: An APM like New Relic would be a tool to see the pods load and response times.
  How services do load balancing is provided here:-https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies
  
  There could be some overhead but not such that two pods provide the same i/o as one.
  
  Could you check if there could be other factors like ingress-controller configurations and such?
Comments: rohatgisanat
Tags: <kubernetes><load-balance><performance>
Title: Kubernetes pods scaling doesnt increase throughput
UserDisplayName: Most Wanted
ThreadText: # Kubernetes pods scaling doesnt increase throughput
  
  (Asked by: username_0 on 2021-07-07)
  
  I have a simple Django web application running on AWS EC2 instance (2 vCPU, 4Gi).
  For that type of deployment I have a performance (measured with [Hey](https://github.com/rakyll/hey))
  
  ```
  $ hey -t 30 -z 1m https://mydomain.com/my/endpoint/
  
  ```
  
  about `30 reqs/s`
  
  When deployed with Kubernetes onto EKS cluster (same docker image) I have about `15 reqs/s` for 1 pod with limits
  
  ```
      resources:
        requests:
          cpu: "500m"
          memory: "2Gi"
        limits:
          cpu: "1"
          memory: "4Gi"
  
  ```
  
  I expect that scaling pods would double the performance but nothing happens (request rate stays the same)
  
  ```
  $ kubectl scale deploy/myapp --replicas=2
  
  ```
  
  I'm running Django app behind `gunicorn` like this (adjusting number of workers did not help)
  
  ```
  gunicorn --config gunicorn.py myapp.wsgi:application
  # gunicorn.py
  import os
  
  # tried different values from 1 to 5
  # for 1 vCPU allocated for the pod it should be 3 as per docs
  # (2 x $num_cores) + 1
  # https://docs.gunicorn.org/en/stable/design.html#how-many-workers
  workers = int(os.getenv('GUNICORN_WORKERS_APP', default=4))
  daemon = False
  bind = '0.0.0.0:8080'
  max_requests = 2000
  
  ```
  
  So couple of questions here:
  
  * how to properly debug this type of issue and what's the toolset I can use?
  * how `service` does load balancing to pods and is there any overhead introduced?
  * is there an overhead for resource management (cpu allocation/throttling)?
  * how performance tuning is done for k8s-driven applications?
  
  **Tags:** kubernetes, load-balance, performance
  
  > **Comments:**
  > **username_1**: An APM like New Relic would be a tool to see the pods load and response times. How services do load balancing is provided here:-https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies  There could be some overhead but not such that two pods provide the same i/o as one.  Could you check if there could be other factors like ingress-controller configurations and such?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-07-15. Score: 1
  
  What does 'kubectl top node' return for your node usage?
  
  Kubernetes is pretty low overhead and can certainly handle more than 15req/s. Does your application rely on a database? If so, where is that database living? Is that single node also running any other pods?
  
  As others noted, an APM would be a helpful tool to diagnose unexpected performance issues.
  
  
  ---
  
  