title: Cloud Endpoints Framework Version 2 inconsistent and high unexplainable latency
body: <p>I recently migrated my App from version 1 of Cloud Endpoints Frameworks to version 2 (App Engine python standard). Supposedly one of the benefits is reduced request latency. Ignoring warm up and/or starting the backend instance, it seems I was getting unexplained latency outside observable logs/app stats, of anywhere from 300ms to 2 seconds. After observing this for about a week I finally just went back to the echo example and deployed to a test app and noticed the exact same behavior.</p>
  
  <p>Echo example: <a href="https://cloud.google.com/endpoints/docs/frameworks/python/get-started-frameworks-python" rel="nofollow noreferrer">https://cloud.google.com/endpoints/docs/frameworks/python/get-started-frameworks-python</a></p>
  
  <p>Here is the relevant instance settings in my app.yaml:</p>
  
  <pre><code>runtime: python27
  env: standard
  threadsafe: true
  instance_class: B2
  basic_scaling:
      idle_timeout: 900s
      max_instances: 2
  </code></pre>
  
  <p>Here are 3 requests to the api via curl spaced multiple seconds apart. The test app should not be doing anything else:</p>
  
  <pre><code>MacBook-Pro$ curl -H "Content-Type: application/json" -X POST -d '{"content":"hello world"}' https://testtictactoe-164905.appspot.com/_ah/api/echo/v1/echo -w "@curl-format.txt"
  {
   "content": "hello world"
  }
  time_namelookup: 0.005
  time_connect: 0.019
  time_appconnect: 0.152
  time_pretransfer: 0.152
  time_redirect: 0.000
  time_starttransfer: 0.263
  --------
  time_total: 0.263
  
  MacBook-Pro$ curl -H "Content-Type: application/json" -X POST -d '{"content":"hello world"}' https://testtictactoe-164905.appspot.com/_ah/api/echo/v1/echo -w "@curl-format.txt"
  {
   "content": "hello world"
  }
  time_namelookup: 0.005
  time_connect: 0.020
  time_appconnect: 0.144
  time_pretransfer: 0.144
  time_redirect: 0.000
  time_starttransfer: 0.613
  --------
  time_total: 0.613
  
  MacBook-Pro$ curl -H "Content-Type: application/json" -X POST -d '{"content":"hello world"}' https://testtictactoe-164905.appspot.com/_ah/api/echo/v1/echo -w "@curl-format.txt"
  {
   "content": "hello world"
  }
  time_namelookup: 0.005
  time_connect: 0.021
  time_appconnect: 0.145
  time_pretransfer: 0.145
  time_redirect: 0.000
  time_starttransfer: 1.028
  --------
  time_total: 1.028
  </code></pre>
  
  <p>Here is the app stats for the first 0.263 second request (Grand Total 48ms):
  <a href="https://i.stack.imgur.com/KfNkc.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/KfNkc.png" alt="enter image description here"></a>
  Here is part of the log entry for 0.263 second request:
  <a href="https://i.stack.imgur.com/6H68V.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/6H68V.png" alt="enter image description here"></a></p>
  
  <p>Here is the app stats for the 1.028 second request (Grand Total 504ms):</p>
  
  <p><a href="https://i.stack.imgur.com/54l4v.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/54l4v.png" alt="App stats for 1.028 second request"></a>
  Here is part of the log entry for the 1.028 second request
  <a href="https://i.stack.imgur.com/8Kqd5.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8Kqd5.png" alt="enter image description here"></a></p>
  
  <p>Here is the instance latency for the last hour:</p>
  
  <p><a href="https://i.stack.imgur.com/LLpOH.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/LLpOH.png" alt="enter image description here"></a></p>
  
  <ol>
  <li>Why is the actual latency of my request so much higher than what is shown by the logs and app stats?</li>
  <li>Is this extra overhead latency of up to 600ms, which seems completely out of my control, just an expected part of making google cloud endpoints requests?</li>
  <li>Why does this latency fluctuate so much when dealing with just one instance and very few incoming requests?</li>
  </ol>