<p dir="auto">From <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="158469029" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/26823" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/26823/hovercard" href="https://github.com/kubernetes/kubernetes/pull/26823">#26823</a> on the 1-2 release branch.</p>
<p dir="auto"><a href="https://console.cloud.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/26823/kubernetes-pull-test-unit-integration/29130/" rel="nofollow">https://console.cloud.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/26823/kubernetes-pull-test-unit-integration/29130/</a></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="=== RUN   TestEtcdDeleteCollectionNotFound
I0603 16:07:09.501076   31650 trace.go:57] Trace &quot;etcdHelper::Create *api.Pod&quot; (started 2016-06-03 16:07:09.021087134 -0700 PDT):
[10.438489ms] [10.438489ms] Object encoded
[12.938103ms] [2.499614ms] Version checked
[478.819054ms] [465.880951ms] Object created
[479.50163ms] [682.576Âµs] END
==================
WARNING: DATA RACE
Write by goroutine 136:
  testing.(*common).FailNow()
      /tmp/workdir/go/src/testing/testing.go:335 +0x41
  testing.(*common).Fatalf()
      /tmp/workdir/go/src/testing/testing.go:379 +0x94
  k8s.io/kubernetes/pkg/registry/generic/etcd.TestEtcdDeleteCollectionNotFound.func1()
      /workspace/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/generic/etcd/etcd_test.go:607 +0x1eb

Previous write by goroutine 127:
  testing.(*common).FailNow()
      /tmp/workdir/go/src/testing/testing.go:335 +0x41
  testing.(*common).Fatalf()
      /tmp/workdir/go/src/testing/testing.go:379 +0x94
  k8s.io/kubernetes/pkg/registry/generic/etcd.TestEtcdDeleteCollectionNotFound.func1()
      /workspace/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/generic/etcd/etcd_test.go:607 +0x1eb

Goroutine 136 (running) created at:
  k8s.io/kubernetes/pkg/registry/generic/etcd.TestEtcdDeleteCollectionNotFound()
      /workspace/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/generic/etcd/etcd_test.go:609 +0x674
  testing.tRunner()
      /tmp/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 127 (finished) created at:
  k8s.io/kubernetes/pkg/registry/generic/etcd.TestEtcdDeleteCollectionNotFound()
      /workspace/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/generic/etcd/etcd_test.go:609 +0x674
  testing.tRunner()
      /tmp/workdir/go/src/testing/testing.go:456 +0xdc
==================
I0603 16:07:31.563128   31650 trace.go:57] Trace &quot;etcdHelper::Create *api.Pod&quot; (started 2016-06-03 16:07:30.904025457 -0700 PDT):
[4.074994ms] [4.074994ms] Object encoded
[4.108285ms] [33.291Âµs] Version checked
[388.137603ms] [384.029318ms] Object created
[658.894085ms] [270.756482ms] END
I0603 16:07:33.703849   31650 trace.go:57] Trace &quot;List *api.PodList&quot; (started 2016-06-03 16:07:31.608403789 -0700 PDT):
[21.447Âµs] [21.447Âµs] About to list etcd node
[1.910621611s] [1.910600164s] Etcd node listed
[2.095241867s] [184.620256ms] Node list decoded
[2.095271215s] [29.348Âµs] END
I0603 16:07:40.311964   31650 trace.go:57] Trace &quot;etcdHelper::Create *api.Pod&quot; (started 2016-06-03 16:07:39.339138415 -0700 PDT):
[8.543508ms] [8.543508ms] Object encoded
[8.57157ms] [28.062Âµs] Version checked
[657.103509ms] [648.531939ms] Object created
[972.615398ms] [315.511889ms] END
I0603 16:07:41.685855   31650 trace.go:57] Trace &quot;List *api.PodList&quot; (started 2016-06-03 16:07:40.451788092 -0700 PDT):
[28.395Âµs] [28.395Âµs] About to list etcd node
[1.233569593s] [1.233541198s] Etcd node listed
[1.23389272s] [323.127Âµs] Node list decoded
[1.233916803s] [24.083Âµs] END
I0603 16:07:41.989831   31650 trace.go:57] Trace &quot;List *api.PodList&quot; (started 2016-06-03 16:07:40.452513447 -0700 PDT):
[17.67Âµs] [17.67Âµs] About to list etcd node
[1.53678103s] [1.53676336s] Etcd node listed
[1.537119905s] [338.875Âµs] Node list decoded
[1.537147305s] [27.4Âµs] END
I0603 16:07:42.136526   31650 trace.go:57] Trace &quot;List *api.PodList&quot; (started 2016-06-03 16:07:40.433819252 -0700 PDT):
[21.971Âµs] [21.971Âµs] About to list etcd node
[1.701880738s] [1.701858767s] Etcd node listed
[1.702409739s] [529.001Âµs] Node list decoded
[1.702443019s] [33.28Âµs] END
I0603 16:07:46.832506   31650 trace.go:57] Trace &quot;etcdHelper::Create *api.Pod&quot; (started 2016-06-03 16:07:46.368422356 -0700 PDT):
[403.833Âµs] [403.833Âµs] Object encoded
[423.064Âµs] [19.231Âµs] Version checked
[463.224621ms] [462.801557ms] Object created
[463.920768ms] [696.147Âµs] END
I0603 16:07:48.421105   31650 trace.go:57] Trace &quot;List *api.PodList&quot; (started 2016-06-03 16:07:46.933007109 -0700 PDT):
[20.505Âµs] [20.505Âµs] About to list etcd node
[1.487499823s] [1.487479318s] Etcd node listed
[1.487894779s] [394.956Âµs] Node list decoded
[1.487921821s] [27.042Âµs] END
I0603 16:07:55.195380   31650 trace.go:57] Trace &quot;etcdHelper::Create *api.Pod&quot; (started 2016-06-03 16:07:54.18723978 -0700 PDT):
[569.053Âµs] [569.053Âµs] Object encoded
[595.08Âµs] [26.027Âµs] Version checked
[1.007338588s] [1.006743508s] Object created
[1.007968516s] [629.928Âµs] END
I0603 16:07:55.848223   31650 trace.go:57] Trace &quot;etcdHelper::Create *api.Pod&quot; (started 2016-06-03 16:07:55.195738587 -0700 PDT):
[265.59Âµs] [265.59Âµs] Object encoded
[283.195Âµs] [17.605Âµs] Version checked
[469.233449ms] [468.950254ms] Object created
[652.176438ms] [182.942989ms] END
I0603 16:07:56.997531   31650 trace.go:57] Trace &quot;List *api.PodList&quot; (started 2016-06-03 16:07:55.904154956 -0700 PDT):
[29.063Âµs] [29.063Âµs] About to list etcd node
[1.092862614s] [1.092833551s] Etcd node listed
[1.093186047s] [323.433Âµs] Node list decoded
[1.093211067s] [25.02Âµs] END
I0603 16:07:57.032214   31650 trace.go:57] Trace &quot;List *api.PodList&quot; (started 2016-06-03 16:07:55.879245406 -0700 PDT):
[37.82Âµs] [37.82Âµs] About to list etcd node
[1.152358883s] [1.152321063s] Etcd node listed
[1.152757988s] [399.105Âµs] Node list decoded
[1.152786584s] [28.596Âµs] END
I0603 16:07:57.421529   31650 trace.go:57] Trace &quot;List *api.PodList&quot; (started 2016-06-03 16:07:55.8518515 -0700 PDT):
[25.787Âµs] [25.787Âµs] About to list etcd node
[1.568348264s] [1.568322477s] Etcd node listed
[1.569206493s] [858.229Âµs] Node list decoded
[1.569266984s] [60.491Âµs] END
I0603 16:07:58.072211   31650 trace.go:57] Trace &quot;List *api.PodList&quot; (started 2016-06-03 16:07:55.853393954 -0700 PDT):
[23.36Âµs] [23.36Âµs] About to list etcd node
[2.218247378s] [2.218224018s] Etcd node listed
[2.218613175s] [365.797Âµs] Node list decoded
[2.218650374s] [37.199Âµs] END
I0603 16:08:04.185381   31650 trace.go:57] Trace &quot;etcdHelper::Create *api.Pod&quot; (started 2016-06-03 16:08:03.844808624 -0700 PDT):
[418.697Âµs] [418.697Âµs] Object encoded
[445.452Âµs] [26.755Âµs] Version checked
[339.711037ms] [339.265585ms] Object created
[340.374126ms] [663.089Âµs] END
--- FAIL: TestEtcdDeleteCollectionNotFound (83.95s)
    etcd_test.go:607: Unexpected error: client: etcd cluster is unavailable or misconfigured
    etcd_test.go:607: Unexpected error: client: etcd cluster is unavailable or misconfigured
    etcd_test.go:607: Unexpected error: client: etcd cluster is unavailable or misconfigured
    etcd_test.go:607: Unexpected error: client: etcd cluster is unavailable or misconfigured
    etcd_test.go:607: Unexpected error: client: etcd cluster is unavailable or misconfigured
    etcd_test.go:614: Unexpected error: &lt;nil&gt;
    etcd_test.go:593: Unexpected error: pods &quot;foo&quot; already exists"><pre class="notranslate"><code class="notranslate">=== RUN   TestEtcdDeleteCollectionNotFound
I0603 16:07:09.501076   31650 trace.go:57] Trace "etcdHelper::Create *api.Pod" (started 2016-06-03 16:07:09.021087134 -0700 PDT):
[10.438489ms] [10.438489ms] Object encoded
[12.938103ms] [2.499614ms] Version checked
[478.819054ms] [465.880951ms] Object created
[479.50163ms] [682.576Âµs] END
==================
WARNING: DATA RACE
Write by goroutine 136:
  testing.(*common).FailNow()
      /tmp/workdir/go/src/testing/testing.go:335 +0x41
  testing.(*common).Fatalf()
      /tmp/workdir/go/src/testing/testing.go:379 +0x94
  k8s.io/kubernetes/pkg/registry/generic/etcd.TestEtcdDeleteCollectionNotFound.func1()
      /workspace/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/generic/etcd/etcd_test.go:607 +0x1eb

Previous write by goroutine 127:
  testing.(*common).FailNow()
      /tmp/workdir/go/src/testing/testing.go:335 +0x41
  testing.(*common).Fatalf()
      /tmp/workdir/go/src/testing/testing.go:379 +0x94
  k8s.io/kubernetes/pkg/registry/generic/etcd.TestEtcdDeleteCollectionNotFound.func1()
      /workspace/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/generic/etcd/etcd_test.go:607 +0x1eb

Goroutine 136 (running) created at:
  k8s.io/kubernetes/pkg/registry/generic/etcd.TestEtcdDeleteCollectionNotFound()
      /workspace/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/generic/etcd/etcd_test.go:609 +0x674
  testing.tRunner()
      /tmp/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 127 (finished) created at:
  k8s.io/kubernetes/pkg/registry/generic/etcd.TestEtcdDeleteCollectionNotFound()
      /workspace/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/generic/etcd/etcd_test.go:609 +0x674
  testing.tRunner()
      /tmp/workdir/go/src/testing/testing.go:456 +0xdc
==================
I0603 16:07:31.563128   31650 trace.go:57] Trace "etcdHelper::Create *api.Pod" (started 2016-06-03 16:07:30.904025457 -0700 PDT):
[4.074994ms] [4.074994ms] Object encoded
[4.108285ms] [33.291Âµs] Version checked
[388.137603ms] [384.029318ms] Object created
[658.894085ms] [270.756482ms] END
I0603 16:07:33.703849   31650 trace.go:57] Trace "List *api.PodList" (started 2016-06-03 16:07:31.608403789 -0700 PDT):
[21.447Âµs] [21.447Âµs] About to list etcd node
[1.910621611s] [1.910600164s] Etcd node listed
[2.095241867s] [184.620256ms] Node list decoded
[2.095271215s] [29.348Âµs] END
I0603 16:07:40.311964   31650 trace.go:57] Trace "etcdHelper::Create *api.Pod" (started 2016-06-03 16:07:39.339138415 -0700 PDT):
[8.543508ms] [8.543508ms] Object encoded
[8.57157ms] [28.062Âµs] Version checked
[657.103509ms] [648.531939ms] Object created
[972.615398ms] [315.511889ms] END
I0603 16:07:41.685855   31650 trace.go:57] Trace "List *api.PodList" (started 2016-06-03 16:07:40.451788092 -0700 PDT):
[28.395Âµs] [28.395Âµs] About to list etcd node
[1.233569593s] [1.233541198s] Etcd node listed
[1.23389272s] [323.127Âµs] Node list decoded
[1.233916803s] [24.083Âµs] END
I0603 16:07:41.989831   31650 trace.go:57] Trace "List *api.PodList" (started 2016-06-03 16:07:40.452513447 -0700 PDT):
[17.67Âµs] [17.67Âµs] About to list etcd node
[1.53678103s] [1.53676336s] Etcd node listed
[1.537119905s] [338.875Âµs] Node list decoded
[1.537147305s] [27.4Âµs] END
I0603 16:07:42.136526   31650 trace.go:57] Trace "List *api.PodList" (started 2016-06-03 16:07:40.433819252 -0700 PDT):
[21.971Âµs] [21.971Âµs] About to list etcd node
[1.701880738s] [1.701858767s] Etcd node listed
[1.702409739s] [529.001Âµs] Node list decoded
[1.702443019s] [33.28Âµs] END
I0603 16:07:46.832506   31650 trace.go:57] Trace "etcdHelper::Create *api.Pod" (started 2016-06-03 16:07:46.368422356 -0700 PDT):
[403.833Âµs] [403.833Âµs] Object encoded
[423.064Âµs] [19.231Âµs] Version checked
[463.224621ms] [462.801557ms] Object created
[463.920768ms] [696.147Âµs] END
I0603 16:07:48.421105   31650 trace.go:57] Trace "List *api.PodList" (started 2016-06-03 16:07:46.933007109 -0700 PDT):
[20.505Âµs] [20.505Âµs] About to list etcd node
[1.487499823s] [1.487479318s] Etcd node listed
[1.487894779s] [394.956Âµs] Node list decoded
[1.487921821s] [27.042Âµs] END
I0603 16:07:55.195380   31650 trace.go:57] Trace "etcdHelper::Create *api.Pod" (started 2016-06-03 16:07:54.18723978 -0700 PDT):
[569.053Âµs] [569.053Âµs] Object encoded
[595.08Âµs] [26.027Âµs] Version checked
[1.007338588s] [1.006743508s] Object created
[1.007968516s] [629.928Âµs] END
I0603 16:07:55.848223   31650 trace.go:57] Trace "etcdHelper::Create *api.Pod" (started 2016-06-03 16:07:55.195738587 -0700 PDT):
[265.59Âµs] [265.59Âµs] Object encoded
[283.195Âµs] [17.605Âµs] Version checked
[469.233449ms] [468.950254ms] Object created
[652.176438ms] [182.942989ms] END
I0603 16:07:56.997531   31650 trace.go:57] Trace "List *api.PodList" (started 2016-06-03 16:07:55.904154956 -0700 PDT):
[29.063Âµs] [29.063Âµs] About to list etcd node
[1.092862614s] [1.092833551s] Etcd node listed
[1.093186047s] [323.433Âµs] Node list decoded
[1.093211067s] [25.02Âµs] END
I0603 16:07:57.032214   31650 trace.go:57] Trace "List *api.PodList" (started 2016-06-03 16:07:55.879245406 -0700 PDT):
[37.82Âµs] [37.82Âµs] About to list etcd node
[1.152358883s] [1.152321063s] Etcd node listed
[1.152757988s] [399.105Âµs] Node list decoded
[1.152786584s] [28.596Âµs] END
I0603 16:07:57.421529   31650 trace.go:57] Trace "List *api.PodList" (started 2016-06-03 16:07:55.8518515 -0700 PDT):
[25.787Âµs] [25.787Âµs] About to list etcd node
[1.568348264s] [1.568322477s] Etcd node listed
[1.569206493s] [858.229Âµs] Node list decoded
[1.569266984s] [60.491Âµs] END
I0603 16:07:58.072211   31650 trace.go:57] Trace "List *api.PodList" (started 2016-06-03 16:07:55.853393954 -0700 PDT):
[23.36Âµs] [23.36Âµs] About to list etcd node
[2.218247378s] [2.218224018s] Etcd node listed
[2.218613175s] [365.797Âµs] Node list decoded
[2.218650374s] [37.199Âµs] END
I0603 16:08:04.185381   31650 trace.go:57] Trace "etcdHelper::Create *api.Pod" (started 2016-06-03 16:08:03.844808624 -0700 PDT):
[418.697Âµs] [418.697Âµs] Object encoded
[445.452Âµs] [26.755Âµs] Version checked
[339.711037ms] [339.265585ms] Object created
[340.374126ms] [663.089Âµs] END
--- FAIL: TestEtcdDeleteCollectionNotFound (83.95s)
    etcd_test.go:607: Unexpected error: client: etcd cluster is unavailable or misconfigured
    etcd_test.go:607: Unexpected error: client: etcd cluster is unavailable or misconfigured
    etcd_test.go:607: Unexpected error: client: etcd cluster is unavailable or misconfigured
    etcd_test.go:607: Unexpected error: client: etcd cluster is unavailable or misconfigured
    etcd_test.go:607: Unexpected error: client: etcd cluster is unavailable or misconfigured
    etcd_test.go:614: Unexpected error: &lt;nil&gt;
    etcd_test.go:593: Unexpected error: pods "foo" already exists
</code></pre></div>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):<br>
No</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<h2 dir="auto">kube-dns, resolv.conf, timeout, skydns</h2>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):<br>
BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):<br>
1.3.6 / kube-dns v17.1</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>:<br>
AWS</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):<br>
centos7</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):</li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
Host resolv.conf, as used in kube-dns container:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="search eu-west-1.compute.internal
nameserver 10.10.224.4
nameserver 10.10.228.4
nameserver 10.10.232.4
options rotate attempts:1 timeout:1
options ndots:5"><pre class="notranslate"><code class="notranslate">search eu-west-1.compute.internal
nameserver 10.10.224.4
nameserver 10.10.228.4
nameserver 10.10.232.4
options rotate attempts:1 timeout:1
options ndots:5
</code></pre></div>
<p dir="auto">If I disable <code class="notranslate">10.10.224.4</code>, every few queries it takes 5s to resolve (resulting in timeouts on other pods). Here's what it looks like when running from the kube-dns pod:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# time nslookup gatech.edu.eu-west-1.compute.internal
nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal'
Command exited with non-zero status 1
real    0m 0.00s
user    0m 0.00s
sys     0m 0.00s
/ # time nslookup gatech.edu.eu-west-1.compute.internal
Server:    10.50.224.4
Address 1: 10.50.224.4

nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal'
Command exited with non-zero status 1
real    0m 5.01s
user    0m 0.00s
sys     0m 0.00s
/ # time nslookup gatech.edu.eu-west-1.compute.internal
Server:    10.50.224.4
Address 1: 10.50.224.4

nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal'
Command exited with non-zero status 1
real    0m 5.00s
user    0m 0.00s
sys     0m 0.00s
/ # time nslookup gatech.edu.eu-west-1.compute.internal
Server:    10.50.224.4
Address 1: 10.50.224.4

nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal'
Command exited with non-zero status 1
real    0m 0.00s
user    0m 0.00s
sys     0m 0.00s
/ # time nslookup gatech.edu.eu-west-1.compute.internal
Server:    10.50.224.4

Address 1: 10.50.224.4

nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal'
Command exited with non-zero status 1
real    0m 5.01s
user    0m 0.00s
sys     0m 0.00s
/ # 
/ # time nslookup gatech.edu.eu-west-1.compute.internal
Server:    10.50.224.4
Address 1: 10.50.224.4

nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal'
Command exited with non-zero status 1
real    0m 0.00s
user    0m 0.00s
sys     0m 0.00s
/ # time nslookup gatech.edu.eu-west-1.compute.internal
Server:    10.50.224.4
Address 1: 10.50.224.4"><pre class="notranslate"><code class="notranslate"># time nslookup gatech.edu.eu-west-1.compute.internal
nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal'
Command exited with non-zero status 1
real    0m 0.00s
user    0m 0.00s
sys     0m 0.00s
/ # time nslookup gatech.edu.eu-west-1.compute.internal
Server:    10.50.224.4
Address 1: 10.50.224.4

nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal'
Command exited with non-zero status 1
real    0m 5.01s
user    0m 0.00s
sys     0m 0.00s
/ # time nslookup gatech.edu.eu-west-1.compute.internal
Server:    10.50.224.4
Address 1: 10.50.224.4

nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal'
Command exited with non-zero status 1
real    0m 5.00s
user    0m 0.00s
sys     0m 0.00s
/ # time nslookup gatech.edu.eu-west-1.compute.internal
Server:    10.50.224.4
Address 1: 10.50.224.4

nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal'
Command exited with non-zero status 1
real    0m 0.00s
user    0m 0.00s
sys     0m 0.00s
/ # time nslookup gatech.edu.eu-west-1.compute.internal
Server:    10.50.224.4

Address 1: 10.50.224.4

nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal'
Command exited with non-zero status 1
real    0m 5.01s
user    0m 0.00s
sys     0m 0.00s
/ # 
/ # time nslookup gatech.edu.eu-west-1.compute.internal
Server:    10.50.224.4
Address 1: 10.50.224.4

nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal'
Command exited with non-zero status 1
real    0m 0.00s
user    0m 0.00s
sys     0m 0.00s
/ # time nslookup gatech.edu.eu-west-1.compute.internal
Server:    10.50.224.4
Address 1: 10.50.224.4
</code></pre></div>
<p dir="auto">And from another pod:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/# time getent ahosts gatech.edu
130.207.160.173 STREAM gatech.edu
130.207.160.173 DGRAM  
130.207.160.173 RAW    

real    0m0.009s
user    0m0.000s
sys     0m0.002s
mypod:/# time getent ahosts gatech.edu
130.207.160.173 STREAM gatech.edu
130.207.160.173 DGRAM  
130.207.160.173 RAW    

real    0m2.007s
user    0m0.000s
sys     0m0.002s
mypod:/# time getent ahosts gatech.edu
130.207.160.173 STREAM gatech.edu
130.207.160.173 DGRAM  
130.207.160.173 RAW    

real    0m0.006s
user    0m0.000s
sys     0m0.002s
mypod:/# time getent ahosts gatech.edu
130.207.160.173 STREAM gatech.edu
130.207.160.173 DGRAM  
130.207.160.173 RAW    

real    0m0.005s
user    0m0.000s
sys     0m0.001s
mypod:/# time getent ahosts gatech.edu
130.207.160.173 STREAM gatech.edu
130.207.160.173 DGRAM  
130.207.160.173 RAW    

real    0m2.008s
user    0m0.000s
sys     0m0.002s"><pre class="notranslate"><code class="notranslate">/# time getent ahosts gatech.edu
130.207.160.173 STREAM gatech.edu
130.207.160.173 DGRAM  
130.207.160.173 RAW    

real    0m0.009s
user    0m0.000s
sys     0m0.002s
mypod:/# time getent ahosts gatech.edu
130.207.160.173 STREAM gatech.edu
130.207.160.173 DGRAM  
130.207.160.173 RAW    

real    0m2.007s
user    0m0.000s
sys     0m0.002s
mypod:/# time getent ahosts gatech.edu
130.207.160.173 STREAM gatech.edu
130.207.160.173 DGRAM  
130.207.160.173 RAW    

real    0m0.006s
user    0m0.000s
sys     0m0.002s
mypod:/# time getent ahosts gatech.edu
130.207.160.173 STREAM gatech.edu
130.207.160.173 DGRAM  
130.207.160.173 RAW    

real    0m0.005s
user    0m0.000s
sys     0m0.001s
mypod:/# time getent ahosts gatech.edu
130.207.160.173 STREAM gatech.edu
130.207.160.173 DGRAM  
130.207.160.173 RAW    

real    0m2.008s
user    0m0.000s
sys     0m0.002s
</code></pre></div>
<p dir="auto">I can sort of work around this by removing the <code class="notranslate">eu-west-1.compute.internal</code> search domain, which prevents the search NXDOMAIN that causes random hangs.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">I expect it to resolve the NXDOMAIN quickly even if one of the nameservers is down. Or at worst, take up to <code class="notranslate">timeout:1</code>. The timeout:1 option seems to be ignored, and it takes 5 seconds to timeout whenever it hits the down server.</p>
<p dir="auto">Also the rotate option seems ignored, so all the load from kube-dns happens on the primary dns server.    (need to verify this, just noticed from quick observation that the primary server gets almost all the queries)</p>
<p dir="auto">Ideally, in my opinion, kube-dns would use something like musl and resolve against all the nameservers in parallel, making it robust to failure.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<ol dir="auto">
<li>Have multiple nameservers in host's resolv.conf</li>
<li>Turn off the first nameserver</li>
<li>Try to resolve domains that would normally return NXDOMAIN</li>
</ol>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>