<p dir="auto">kubernetes-pull-build-test-e2e-gce/33760 for PR <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="143373531" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/23454" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/23454/hovercard" href="https://github.com/kubernetes/kubernetes/pull/23454">#23454</a></p>
<p dir="auto">This might a duplicate of <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="139104616" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/22671" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/22671/hovercard" href="https://github.com/kubernetes/kubernetes/issues/22671">#22671</a>, but the failure wasn't entirely the same.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Stacktrace

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:227
Expected error:
    &lt;*errors.errorString | 0xc20851ad50&gt;: {
        s: &quot;Error running &amp;{/var/lib/jenkins/workspace/kubernetes-pull-build-test-e2e-gce@2/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://146.148.99.243 --kubeconfig=/var/lib/jenkins/workspace/kubernetes-pull-build-test-e2e-gce@2/.kube/config exec --namespace=e2e-tests-kubectl-4sutt -i nginx bash] []  0xc20846b088 hi\n  [] &lt;nil&gt; 0xc208925b20 exit status 2 &lt;nil&gt; true [0xc20846b088 0xc20846b0d0 0xc20846b0f0] [0xc20846b0d0 0xc20846b0f0] [0xc20846b0c8 0xc20846b0e8] [0x972bb0 0x972bb0] 0xc208920d20}:\nCommand stdout:\nhi\n\nstderr:\n\nerror:\nexit status 2\n&quot;,
    }
    Error running &amp;{/var/lib/jenkins/workspace/kubernetes-pull-build-test-e2e-gce@2/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://146.148.99.243 --kubeconfig=/var/lib/jenkins/workspace/kubernetes-pull-build-test-e2e-gce@2/.kube/config exec --namespace=e2e-tests-kubectl-4sutt -i nginx bash] []  0xc20846b088 hi
      [] &lt;nil&gt; 0xc208925b20 exit status 2 &lt;nil&gt; true [0xc20846b088 0xc20846b0d0 0xc20846b0f0] [0xc20846b0d0 0xc20846b0f0] [0xc20846b0c8 0xc20846b0e8] [0x972bb0 0x972bb0] 0xc208920d20}:
    Command stdout:
    hi

    stderr:

    error:
    exit status 2

not to have occurred"><pre class="notranslate"><code class="notranslate">Stacktrace

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:227
Expected error:
    &lt;*errors.errorString | 0xc20851ad50&gt;: {
        s: "Error running &amp;{/var/lib/jenkins/workspace/kubernetes-pull-build-test-e2e-gce@2/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://146.148.99.243 --kubeconfig=/var/lib/jenkins/workspace/kubernetes-pull-build-test-e2e-gce@2/.kube/config exec --namespace=e2e-tests-kubectl-4sutt -i nginx bash] []  0xc20846b088 hi\n  [] &lt;nil&gt; 0xc208925b20 exit status 2 &lt;nil&gt; true [0xc20846b088 0xc20846b0d0 0xc20846b0f0] [0xc20846b0d0 0xc20846b0f0] [0xc20846b0c8 0xc20846b0e8] [0x972bb0 0x972bb0] 0xc208920d20}:\nCommand stdout:\nhi\n\nstderr:\n\nerror:\nexit status 2\n",
    }
    Error running &amp;{/var/lib/jenkins/workspace/kubernetes-pull-build-test-e2e-gce@2/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://146.148.99.243 --kubeconfig=/var/lib/jenkins/workspace/kubernetes-pull-build-test-e2e-gce@2/.kube/config exec --namespace=e2e-tests-kubectl-4sutt -i nginx bash] []  0xc20846b088 hi
      [] &lt;nil&gt; 0xc208925b20 exit status 2 &lt;nil&gt; true [0xc20846b088 0xc20846b0d0 0xc20846b0f0] [0xc20846b0d0 0xc20846b0f0] [0xc20846b0c8 0xc20846b0e8] [0x972bb0 0x972bb0] 0xc208920d20}:
    Command stdout:
    hi

    stderr:

    error:
    exit status 2

not to have occurred
</code></pre></div>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):<br>
1.4.0</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: bare metal</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): all nodes have ubuntu 16.04</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):</li>
<li><strong>Install tools</strong>: manual</li>
<li><strong>Others</strong>: quay.io/coreos/flannel:v0.6.2 VXLAN, but same issue is present with 0.5.5 VXLAN</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
clients gets timeout when contacting the default ingress controller ,<br>
Took me a while to find the cause of a much higher level issue that I had to investigate ( very sporadic timeout from applications behind ingress-controller ) ..<br>
everything now is quite reproducible,<br>
I have 5 nodes cluster, 3 of them are master ( also schedulable ),I'm using flannel networking on bare metals,<br>
The setup is weird and has been create only for test purpose</p>
<ul dir="auto">
<li>I have a "gcr.io/google_containers/serve_hostname" (call it "pod-hostnames" )</li>
<li>a service pointing to pod-hostnames, and the service expose on port 90 to "externalIPs"</li>
<li>then i have created a pod with ( same configuration of "default nginx ingress controller" but slurped configuration, removed the geolocation part that and applied the configuration to standard nginx image "nginx:1.11.4" , so the upstream is static and pointing to the pod-hostnames:port , configuration isn't refreshed automatically ( only manually ) , this pod of this nginx instance is also connected to "hostNetwork: true" ( this has been done on purpose for testing ) and on port 89, so no service required</li>
</ul>
<p dir="auto">everything works as expected,<br>
from multiple clients I can access port 90 or 89 and get the same service..</p>
<p dir="auto">now there is some kind of filter that I didn't understood,<br>
Let's call</p>
<ul dir="auto">
<li>TESTSRVC: "ab -n 1000 -c 10 http://$host:90"     ( which is the test on the nodePort exposed by the service ) NB those are only 1000 requests with concurrency 10!</li>
<li>CURLSRVC: "curl http://$host:90"</li>
<li>TESTNGINX: "ab -n 1000 -c 10 http://$host:89"   ( which is the test on pod, fully exposed with the "hostNetwork: true" flag )  NB those are only 1000 requests with concurrency 10</li>
<li>CURLNGINX: "curl http://$host:89"</li>
</ul>
<p dir="auto">I have 2 pc in office ( pc1 and pc2 ) same external ip<br>
I also have other severs around to use as clients..  ( call them ext1...n)</p>
<p dir="auto">all bare metal nodes in the cluster are in another datacenter , every server have it's dedicated ipv4 ( and are not natted )</p>
<p dir="auto">iptables have "accept" in filter (input/forward) tables policies , no rate limiters, or other special filter ,</p>
<p dir="auto">CURLNGINX works on pc1,pc2,ext1<br>
CURLSRVC works only on pc1,ext2 -that's weird..-</p>
<p dir="auto">if from pc1 I run TESTNGINX ( which takes about 5 seconds to run ) , pc2 cannot connect to nginx for 60 seconds, nginx logs nothing about those requests from pc2, tcpdump see the requests from the pc2, after those 60 seconds.. nginx start again to reply to pc2 , pc1 and ext1 during those 60 seconds have no problem contacting both :89 or :90</p>
<p dir="auto">if from ext1 I run TESTNGINX, there are no issues for ext1,pc1,pc2,ext2<br>
if from ext2 I run TESTNGINX, there are no issues for ext1,pc1,pc2,ext2</p>
<p dir="auto">if I run TESTSRVC from pc1 or ext2, the nginx start logging issues  (499 CLIENT CLOSED REQUEST)  and in his epoll like</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="10.200.9.1 - [10.200.9.1] - - [03/Oct/2016:16:23:35 +0000] &quot;GET /?w2 HTTP/1.1&quot; 499 0 &quot;-&quot; &quot;curl/7.47.0&quot; 97 4.814 10.200.78.7:9376 0 - -
2016/10/03 16:19:51 [info] 14#14: *131367 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while connecting to upstream, client: 10.200.9.1, server: _, request: &quot;GET /?w2 HTTP/1.1&quot;, upstream: &quot;http://10.200.78.7:9376/?w2&quot;, host: &quot;vpr4.xxxxx:89&quot;"><pre class="notranslate"><code class="notranslate">10.200.9.1 - [10.200.9.1] - - [03/Oct/2016:16:23:35 +0000] "GET /?w2 HTTP/1.1" 499 0 "-" "curl/7.47.0" 97 4.814 10.200.78.7:9376 0 - -
2016/10/03 16:19:51 [info] 14#14: *131367 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while connecting to upstream, client: 10.200.9.1, server: _, request: "GET /?w2 HTTP/1.1", upstream: "http://10.200.78.7:9376/?w2", host: "vpr4.xxxxx:89"
</code></pre></div>
<p dir="auto">and no one of the client can reach the port :89, but the port :90 still being available to pc1,ext2  -that's weird too-<br>
seems that some kind of limits has been reached on the service and also to node cannot reach the service .. for 60 seconds..<br>
after 60 seconds.. everything is available again..</p>
<p dir="auto">now.. these 60 seconds seems to be a constant.. is ALWAYS EXACTLY 60 seconds..<br>
I've tryed to search for those 60 seconds.. increase in sysctl all timeout that have 60 seconds or 60.000 ms as limits .. doubled them to 120 seconds but my timeout issues still lasting 60 seconds..</p>
<p dir="auto"><strong>obviously no pods has been restarted (nor manually or automatically) during those 60 seconds , everything that has been tested here have 0 restart.</strong></p>
<p dir="auto">do you have any idea about what I can try to do ?<br>
in your clusters you have the same issue ?</p>