<p dir="auto">Hi,</p>
<p dir="auto">Recenly, one of our clusters on GCE (1.1.7) started having intermittent connection issues. It turned out to be dns resolver issues. Here are my findings during a recent occurance:</p>
<ul dir="auto">
<li>Pods can not resolve anything.</li>
<li>If I remove skydns ip from /etc/resolv.conf, they work fine. (secondary ip is 169.254.169.254)</li>
<li>skydns logs <code class="notranslate">2016/02/29 04:54:22 skydns: failure to forward request "read udp 169.254.169.254:53: i/o timeout"</code> lines during these outages.</li>
<li>Our apps perform a lot of requests on several google apis and facebook apis. These would create a lot of dns requests.</li>
<li>Using default config provided by google cloud.</li>
<li>Tried replacing all nodes, which effectively restarted every single pod and this did not resolve anything.</li>
<li>Created another cluster while this was happening, it came up with the same issues. This would make me believe that the dns server was down but only skydns was failing at this point.</li>
</ul>
<p dir="auto">I thought maybe the dns server was throttling our requests. This would make sense that multiple clusters failed at the same time. But being able to directly query <code class="notranslate">169.254.169.254</code> from pods seems to eliminate this.</p>
<p dir="auto">So, for some reason, skydns starts failing and recovers by itself. On the host machines everything works perfectly fine as they query the GCE dns server directly.</p>
<p dir="auto">What could be the reason / solution in this?</p>
<p dir="auto">Thanks!</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/.):no" rel="nofollow">http://kubernetes.io/docs/troubleshooting/.):no</a></p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): create,service</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):<br>
FEATURE REQUEST</p>

<p dir="auto"><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1378011" rel="nofollow">https://bugzilla.redhat.com/show_bug.cgi?id=1378011</a><br>
<strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>:</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):</li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>