<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;4+&quot;, GitVersion:&quot;v1.4.0-beta.8&quot;, GitCommit:&quot;3040f87c570a772ce94349b379f41f329494a4f7&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-09-18T21:06:37Z&quot;, GoVersion:&quot;go1.6.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;4+&quot;, GitVersion:&quot;v1.4.0-beta.8&quot;, GitCommit:&quot;3040f87c570a772ce94349b379f41f329494a4f7&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-09-18T21:00:36Z&quot;, GoVersion:&quot;go1.6.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Client Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-beta.8", GitCommit:"3040f87c570a772ce94349b379f41f329494a4f7", GitTreeState:"clean", BuildDate:"2016-09-18T21:06:37Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-beta.8", GitCommit:"3040f87c570a772ce94349b379f41f329494a4f7", GitTreeState:"clean", BuildDate:"2016-09-18T21:00:36Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: GCE+CoreOS+custom k8s</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): CoreOS</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): 4.6.3-coreos</li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
Could not access service via external LB from a kubernetes pod on the node that does not host a pod for that service.</p>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
Traffic should have gone to the external LB which would pick the node that is running a pod for that service.</p>
<p dir="auto"><strong>TL;DR</strong> External load balancer kube-proxy iptable rules should only match on traffic originating from outside the cluster/pod networks i.e. from the external load balancer.<br>
Otherwise traffic originating from within k8s cluster to external load balanced ip addresses can get blackholed.</p>
<p dir="auto">We are having an other issue with <code class="notranslate">OnlyLocal</code>.<br>
I have a service with the following definition</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
   &quot;apiVersion&quot;: &quot;v1&quot;,
   &quot;kind&quot;: &quot;Service&quot;,
   &quot;metadata&quot;: {
      &quot;annotations&quot;: {
         &quot;service.alpha.kubernetes.io/external-traffic&quot;: &quot;OnlyLocal&quot;
      },
      &quot;labels&quot;: {
         &quot;k8s-app&quot;: &quot;my-app&quot;,
         &quot;kubernetes.io/cluster-service&quot;: &quot;true&quot;
      },
      &quot;name&quot;: &quot;my-app&quot;,
      &quot;namespace&quot;: &quot;default&quot;
   },
   &quot;spec&quot;: {
      &quot;loadBalancerIP&quot;: &quot;1.2.3.4&quot;,
      &quot;ports&quot;: [
         {
            &quot;name&quot;: &quot;http&quot;,
            &quot;port&quot;: 80,
            &quot;targetPort&quot;: 80
         }
      ],
      &quot;selector&quot;: {
         &quot;k8s-app&quot;: &quot;my-app&quot;
      },
      &quot;type&quot;: &quot;LoadBalancer&quot;
   }
}"><pre class="notranslate"><code class="notranslate">{
   "apiVersion": "v1",
   "kind": "Service",
   "metadata": {
      "annotations": {
         "service.alpha.kubernetes.io/external-traffic": "OnlyLocal"
      },
      "labels": {
         "k8s-app": "my-app",
         "kubernetes.io/cluster-service": "true"
      },
      "name": "my-app",
      "namespace": "default"
   },
   "spec": {
      "loadBalancerIP": "1.2.3.4",
      "ports": [
         {
            "name": "http",
            "port": 80,
            "targetPort": 80
         }
      ],
      "selector": {
         "k8s-app": "my-app"
      },
      "type": "LoadBalancer"
   }
}
</code></pre></div>
<p dir="auto">If I run</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl http://1.2.3.4/test "><pre class="notranslate"><code class="notranslate">curl http://1.2.3.4/test 
</code></pre></div>
<p dir="auto">From my local box that is outside of the k8s cluster I get a response.</p>
<p dir="auto">When I run the same command from a k8s node that is not the node that is hosting the pod for <code class="notranslate">my-app</code> I get no response.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl http://1.2.3.4/test -v
*   Trying 1.2.3.4
"><pre class="notranslate"><code class="notranslate">curl http://1.2.3.4/test -v
*   Trying 1.2.3.4

</code></pre></div>
<p dir="auto">Checking the iptables rules. Some stuff truncated:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="-A KUBE-SERVICES -d 1.2.3.4/32 -p tcp -m comment --comment &quot;default/my-app:http loadbalancer IP&quot; -m tcp --dport 80 -j KUBE-FW-ABZR2FBH2NLOJM2K
-A KUBE-FW-ABZR2FBH2NLOJM2K -m comment --comment &quot;default/my-app:http loadbalancer IP&quot; -j KUBE-XLB-ABZR2FBH2NLOJM2K
-A KUBE-XLB-ABZR2FBH2NLOJM2K -m comment --comment &quot;default/my-app:http has no local endpoints&quot; -j KUBE-MARK-DROP"><pre class="notranslate"><code class="notranslate">-A KUBE-SERVICES -d 1.2.3.4/32 -p tcp -m comment --comment "default/my-app:http loadbalancer IP" -m tcp --dport 80 -j KUBE-FW-ABZR2FBH2NLOJM2K
-A KUBE-FW-ABZR2FBH2NLOJM2K -m comment --comment "default/my-app:http loadbalancer IP" -j KUBE-XLB-ABZR2FBH2NLOJM2K
-A KUBE-XLB-ABZR2FBH2NLOJM2K -m comment --comment "default/my-app:http has no local endpoints" -j KUBE-MARK-DROP
</code></pre></div>
<p dir="auto">So it looks like the traffic gets dropped by the <code class="notranslate">kube-proxy</code>.<br>
The <code class="notranslate">KUBE-SERVICES</code> seems to match packets that originate from pods on this machine and not only external traffic. This means that if a pod on this machine wants to call a service through the external LB<br>
(e.g. when using external DNS registrations of services) the traffic will be blackholed if the given node does not have a pod in question.<br>
Proposed solution would be to augment the <code class="notranslate">KUBE-SERVICES</code> rule to filter out traffic originating from clusters/pods private network.</p>
<p dir="auto">This is related to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="166941466" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/29409" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/29409/hovercard" href="https://github.com/kubernetes/kubernetes/pull/29409">#29409</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/girishkalele/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/girishkalele">@girishkalele</a></p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto">No</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<p dir="auto">watch leak</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>
<p dir="auto">BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto">1.4.0</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: vagrant on Mac OS X</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): CoreOS 1164.1.0</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):  <code class="notranslate">Linux core-01 4.7.3-coreos #1 SMP Sat Sep 10 08:21:12 UTC 2016 x86_64 Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz GenuineIntel GNU/Linux</code></li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">We ran controller which handles our own ThirdPartyResource.<br>
When we hit "watch of ... ended with: 401: The event in requested index is outdated and cleared (the requested history has been cleared" error, "forcing resync" log is duplicated.<br>
This log message comes from <a href="https://github.com/kubernetes/kubernetes/blob/1d323adade5fc6ecd898908d2c6cb391c8748835/pkg/client/cache/reflector.go#L284">here</a>.  It looks like this goroutine does not exit when we hit the above mentioned error, and continued to work, and running "resync" more frequently.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">The resync goroutine must be terminated when watch is closed (i.e., ListAndWatch function returned), instead of leaking it.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">I just run our controller for ~15 minutes.  I'm not sure why "401: The event in requested index is outdated and cleared (the requested history has been cleared" happens, but from the issues I read, it should be harmless.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>