Answers: <p>Google Support tracked down the issue down for us:</p>
  <blockquote>
  <p>this issue appears to have been caused by a known Issue affecting
  Konnectivity, under some circumstances a memory leak could occur, in
  the case of your cluster, this appears to end up causing
  connection timeouts after some time, this issue has already been
  addressed on the Konnectivity network proxy [1], and the fix is
  already available within GKE on version 1.19.9 or higher.</p>
  </blockquote>
  <p>[1] <a href="https://github.com/kubernetes-sigs/apiserver-network-proxy/pull/168" rel="nofollow noreferrer">https://github.com/kubernetes-sigs/apiserver-network-proxy/pull/168</a></p>
  
Answers: CC BY-SA 4.0
Answers: 2021-05-04T08:44:55.203Z
Answers: 2021-05-04T08:44:55.203Z
Answers: Matt Russell
Body: <p>We have an issue in our GKE cluster that started suddenly yesterday where intermittently we get errors interacting with the control plane. This affects our Jenkins builds which use the Kubernetes plugin, but also even simply running and attaching to pods. For example, if we do a <code>kubectl run -i --tty --rm --image=busybox some-name</code>, about 20% of the time we'll get the following error:</p>
  <pre><code>Error attaching, falling back to logs: error dialing backend: dial timeout
  </code></pre>
  <p>Hoping someone can help suggest how we can diagnose further.</p>
  <p>Verbose logs from the <code>kubectl run</code>:</p>
  <pre><code>I0421 09:38:47.720435    2483 round_trippers.go:420] POST https://35.189.86.41/api/v1/namespaces/default/pods
  I0421 09:38:47.720448    2483 round_trippers.go:427] Request Headers:
  I0421 09:38:47.720452    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
  I0421 09:38:47.720456    2483 round_trippers.go:431]     Content-Type: application/json
  I0421 09:38:47.720460    2483 round_trippers.go:431]     Accept: application/json, */*
  I0421 09:38:47.852510    2483 round_trippers.go:446] Response Status: 201 Created in 132 milliseconds
  I0421 09:38:47.860462    2483 reflector.go:175] Starting reflector *v1.Pod (0s) from k8s.io/client-go/tools/watch/informerwatcher.go:146
  I0421 09:38:47.860489    2483 reflector.go:211] Listing and watching *v1.Pod from k8s.io/client-go/tools/watch/informerwatcher.go:146
  I0421 09:38:47.860610    2483 round_trippers.go:420] GET https://35.189.86.41/api/v1/namespaces/default/pods?fieldSelector=metadata.name%3Dsome-name&amp;limit=500&amp;resourceVersion=0
  I0421 09:38:47.860619    2483 round_trippers.go:427] Request Headers:
  I0421 09:38:47.860628    2483 round_trippers.go:431]     Accept: application/json, */*
  I0421 09:38:47.860633    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
  I0421 09:38:47.912301    2483 round_trippers.go:446] Response Status: 200 OK in 51 milliseconds
  I0421 09:38:47.919196    2483 round_trippers.go:420] GET https://35.189.86.41/api/v1/namespaces/default/pods?allowWatchBookmarks=true&amp;fieldSelector=metadata.name%3Dsome-name&amp;resourceVersion=298429547&amp;timeout=6m8s&amp;timeoutSeconds=368&amp;watch=true
  I0421 09:38:47.919212    2483 round_trippers.go:427] Request Headers:
  I0421 09:38:47.919219    2483 round_trippers.go:431]     Accept: application/json, */*
  I0421 09:38:47.919224    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
  I0421 09:38:47.964479    2483 shared_informer.go:253] caches populated
  I0421 09:38:48.002537    2483 round_trippers.go:446] Response Status: 200 OK in 83 milliseconds
  I0421 09:38:49.416988    2483 reflector.go:181] Stopping reflector *v1.Pod (0s) from k8s.io/client-go/tools/watch/informerwatcher.go:146
  I0421 09:38:49.417115    2483 attach.go:338] defaulting container name to some-name
  If you don't see a command prompt, try pressing enter.
  I0421 09:38:49.417419    2483 round_trippers.go:420] POST https://35.189.86.41/api/v1/namespaces/default/pods/some-name/attach?container=some-name&amp;stdin=true&amp;stdout=true&amp;tty=true
  I0421 09:38:49.417431    2483 round_trippers.go:427] Request Headers:
  I0421 09:38:49.417436    2483 round_trippers.go:431]     X-Stream-Protocol-Version: v4.channel.k8s.io
  I0421 09:38:49.417440    2483 round_trippers.go:431]     X-Stream-Protocol-Version: v3.channel.k8s.io
  I0421 09:38:49.417444    2483 round_trippers.go:431]     X-Stream-Protocol-Version: v2.channel.k8s.io
  I0421 09:38:49.417447    2483 round_trippers.go:431]     X-Stream-Protocol-Version: channel.k8s.io
  I0421 09:38:49.417451    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
  I0421 09:39:19.524483    2483 round_trippers.go:446] Response Status: 500 Internal Server Error in 30106 milliseconds
  Error attaching, falling back to logs: error dialing backend: dial timeout
  I0421 09:39:19.525463    2483 attach.go:338] defaulting container name to some-name
  I0421 09:39:19.525813    2483 round_trippers.go:420] GET https://35.189.86.41/api/v1/namespaces/default/pods/some-name/log?container=some-name
  I0421 09:39:19.525829    2483 round_trippers.go:427] Request Headers:
  I0421 09:39:19.525837    2483 round_trippers.go:431]     Accept: application/json, */*
  I0421 09:39:19.525843    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
  I0421 09:39:49.585194    2483 round_trippers.go:446] Response Status: 500 Internal Server Error in 30059 milliseconds
  I0421 09:39:49.591557    2483 round_trippers.go:420] GET https://35.189.86.41/api/v1/namespaces/default/pods/some-name
  I0421 09:39:49.591578    2483 round_trippers.go:427] Request Headers:
  I0421 09:39:49.591585    2483 round_trippers.go:431]     Accept: application/json, */*
  I0421 09:39:49.591611    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
  I0421 09:39:49.636553    2483 round_trippers.go:446] Response Status: 200 OK in 44 milliseconds
  I0421 09:39:49.639346    2483 round_trippers.go:420] DELETE https://35.189.86.41/api/v1/namespaces/default/pods/some-name
  I0421 09:39:49.639366    2483 round_trippers.go:427] Request Headers:
  I0421 09:39:49.639373    2483 round_trippers.go:431]     Accept: application/json, */*
  I0421 09:39:49.639379    2483 round_trippers.go:431]     Content-Type: application/json
  I0421 09:39:49.639386    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
  I0421 09:39:49.692164    2483 round_trippers.go:446] Response Status: 200 OK in 52 milliseconds
  pod &quot;some-name&quot; deleted
  I0421 09:39:49.692460    2483 helpers.go:216] server response object: [{
    &quot;metadata&quot;: {},
    &quot;status&quot;: &quot;Failure&quot;,
    &quot;message&quot;: &quot;Get \&quot;https://10.31.60.49:10250/containerLogs/default/some-name/some-name\&quot;: dial timeout&quot;,
    &quot;code&quot;: 500
  }]
  F0421 09:39:49.692479    2483 helpers.go:115] Error from server: Get &quot;https://10.31.60.49:10250/containerLogs/default/some-name/some-name&quot;: dial timeout
  </code></pre>
  
Tags: <kubernetes><google-cloud-platform>
Title: How to diagnose intermittent "error dialing backend: dial timeout" error in GKE
UserDisplayName: Matt Russell
ThreadText: # How to diagnose intermittent "error dialing backend: dial timeout" error in GKE
  
  (Asked by: username_0 on 2021-04-21)
  
  We have an issue in our GKE cluster that started suddenly yesterday where intermittently we get errors interacting with the control plane. This affects our Jenkins builds which use the Kubernetes plugin, but also even simply running and attaching to pods. For example, if we do a `kubectl run -i --tty --rm --image=busybox some-name`, about 20% of the time we'll get the following error:
  
  ```
  Error attaching, falling back to logs: error dialing backend: dial timeout
  
  ```
  
  Hoping someone can help suggest how we can diagnose further.
  
  Verbose logs from the `kubectl run`:
  
  ```
  I0421 09:38:47.720435    2483 round_trippers.go:420] POST https://35.189.86.41/api/v1/namespaces/default/pods
  I0421 09:38:47.720448    2483 round_trippers.go:427] Request Headers:
  I0421 09:38:47.720452    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
  I0421 09:38:47.720456    2483 round_trippers.go:431]     Content-Type: application/json
  I0421 09:38:47.720460    2483 round_trippers.go:431]     Accept: application/json, */*
  I0421 09:38:47.852510    2483 round_trippers.go:446] Response Status: 201 Created in 132 milliseconds
  I0421 09:38:47.860462    2483 reflector.go:175] Starting reflector *v1.Pod (0s) from k8s.io/client-go/tools/watch/informerwatcher.go:146
  I0421 09:38:47.860489    2483 reflector.go:211] Listing and watching *v1.Pod from k8s.io/client-go/tools/watch/informerwatcher.go:146
  I0421 09:38:47.860610    2483 round_trippers.go:420] GET https://35.189.86.41/api/v1/namespaces/default/pods?fieldSelector=metadata.name%3Dsome-name&limit=500&resourceVersion=0
  I0421 09:38:47.860619    2483 round_trippers.go:427] Request Headers:
  I0421 09:38:47.860628    2483 round_trippers.go:431]     Accept: application/json, */*
  I0421 09:38:47.860633    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
  I0421 09:38:47.912301    2483 round_trippers.go:446] Response Status: 200 OK in 51 milliseconds
  I0421 09:38:47.919196    2483 round_trippers.go:420] GET https://35.189.86.41/api/v1/namespaces/default/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name&resourceVersion=298429547&timeout=6m8s&timeoutSeconds=368&watch=true
  I0421 09:38:47.919212    2483 round_trippers.go:427] Request Headers:
  I0421 09:38:47.919219    2483 round_trippers.go:431]     Accept: application/json, */*
  I0421 09:38:47.919224    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
  I0421 09:38:47.964479    2483 shared_informer.go:253] caches populated
  I0421 09:38:48.002537    2483 round_trippers.go:446] Response Status: 200 OK in 83 milliseconds
  I0421 09:38:49.416988    2483 reflector.go:181] Stopping reflector *v1.Pod (0s) from k8s.io/client-go/tools/watch/informerwatcher.go:146
  I0421 09:38:49.417115    2483 attach.go:338] defaulting container name to some-name
  If you don't see a command prompt, try pressing enter.
  I0421 09:38:49.417419    2483 round_trippers.go:420] POST https://35.189.86.41/api/v1/namespaces/default/pods/some-name/attach?container=some-name&stdin=true&stdout=true&tty=true
  I0421 09:38:49.417431    2483 round_trippers.go:427] Request Headers:
  I0421 09:38:49.417436    2483 round_trippers.go:431]     X-Stream-Protocol-Version: v4.channel.k8s.io
  I0421 09:38:49.417440    2483 round_trippers.go:431]     X-Stream-Protocol-Version: v3.channel.k8s.io
  I0421 09:38:49.417444    2483 round_trippers.go:431]     X-Stream-Protocol-Version: v2.channel.k8s.io
  I0421 09:38:49.417447    2483 round_trippers.go:431]     X-Stream-Protocol-Version: channel.k8s.io
  I0421 09:38:49.417451    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
  I0421 09:39:19.524483    2483 round_trippers.go:446] Response Status: 500 Internal Server Error in 30106 milliseconds
  Error attaching, falling back to logs: error dialing backend: dial timeout
  I0421 09:39:19.525463    2483 attach.go:338] defaulting container name to some-name
  I0421 09:39:19.525813    2483 round_trippers.go:420] GET https://35.189.86.41/api/v1/namespaces/default/pods/some-name/log?container=some-name
  I0421 09:39:19.525829    2483 round_trippers.go:427] Request Headers:
  I0421 09:39:19.525837    2483 round_trippers.go:431]     Accept: application/json, */*
  I0421 09:39:19.525843    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
  I0421 09:39:49.585194    2483 round_trippers.go:446] Response Status: 500 Internal Server Error in 30059 milliseconds
  I0421 09:39:49.591557    2483 round_trippers.go:420] GET https://35.189.86.41/api/v1/namespaces/default/pods/some-name
  I0421 09:39:49.591578    2483 round_trippers.go:427] Request Headers:
  I0421 09:39:49.591585    2483 round_trippers.go:431]     Accept: application/json, */*
  I0421 09:39:49.591611    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
  I0421 09:39:49.636553    2483 round_trippers.go:446] Response Status: 200 OK in 44 milliseconds
  I0421 09:39:49.639346    2483 round_trippers.go:420] DELETE https://35.189.86.41/api/v1/namespaces/default/pods/some-name
  I0421 09:39:49.639366    2483 round_trippers.go:427] Request Headers:
  I0421 09:39:49.639373    2483 round_trippers.go:431]     Accept: application/json, */*
  I0421 09:39:49.639379    2483 round_trippers.go:431]     Content-Type: application/json
  I0421 09:39:49.639386    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
  I0421 09:39:49.692164    2483 round_trippers.go:446] Response Status: 200 OK in 52 milliseconds
  pod "some-name" deleted
  I0421 09:39:49.692460    2483 helpers.go:216] server response object: [{
    "metadata": {},
    "status": "Failure",
    "message": "Get \"https://10.31.60.49:10250/containerLogs/default/some-name/some-name\": dial timeout",
    "code": 500
  }]
  F0421 09:39:49.692479    2483 helpers.go:115] Error from server: Get "https://10.31.60.49:10250/containerLogs/default/some-name/some-name": dial timeout
  
  ```
  
  **Tags:** kubernetes, google-cloud-platform
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-05-04. Score: 1
  
  Google Support tracked down the issue down for us:
  
  > this issue appears to have been caused by a known Issue affecting
  > Konnectivity, under some circumstances a memory leak could occur, in
  > the case of your cluster, this appears to end up causing
  > connection timeouts after some time, this issue has already been
  > addressed on the Konnectivity network proxy [1], and the fix is
  > already available within GKE on version 1.19.9 or higher.
  
  [1] <https://github.com/kubernetes-sigs/apiserver-network-proxy/pull/168>
  
  
  ---
  
  