<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.): no.</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): documentation, yaml, hack/generate-docs</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): BUG REPORT</p>
<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>): 1.4.5</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: local install</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): CentOS 7</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):3.10.0-327.28.3.el7.x86_64</li>
<li><strong>Install tools</strong>: manual</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
Download 1.4.5 release tarball from Github:<br>
<a href="https://github.com/kubernetes/kubernetes/releases/download/v1.4.5/kubernetes.tar.gz">https://github.com/kubernetes/kubernetes/releases/download/v1.4.5/kubernetes.tar.gz</a><br>
Unpack it and look at documentation folder docs/yaml/kubectl/...<br>
Inside that folder user can see bunch of yaml files that have content:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="This file is autogenerated, but we've stopped checking such files into the
repository to reduce the need for rebases. Please run hack/generate-docs.sh to
populate this file."><pre class="notranslate"><code class="notranslate">This file is autogenerated, but we've stopped checking such files into the
repository to reduce the need for rebases. Please run hack/generate-docs.sh to
populate this file.
</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
In my opinion, those placeholders should be replaced with real generated documentation at the time of release. So, somewhere in release scripts call to generate documentation is missing.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<div class="highlight highlight-text-shell-session notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ wget https://github.com/kubernetes/kubernetes/releases/download/v1.4.5/kubernetes.tar.gz
$ tar zxf kubernetes.tar.gz
$ cat kubernetes/docs/yaml/kubectl/kubectl_api-versions.yaml"><pre class="notranslate">$ <span class="pl-s1">wget https://github.com/kubernetes/kubernetes/releases/download/v1.4.5/kubernetes.tar.gz</span>
$ <span class="pl-s1">tar zxf kubernetes.tar.gz</span>
$ <span class="pl-s1">cat kubernetes/docs/yaml/kubectl/kubectl_api-versions.yaml</span></pre></div>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">I noticed that while connections between pods were usually nearly instant, sometimes they took a few seconds and they sometimes hung altogether, so I started poking around. I quickly found that <code class="notranslate">kube-proxy</code> is taking up between 80% - 100% of a CPU on each system that it's running on. The logs weren't very interesting, but then I checked the iptables rules:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# sudo iptables -L -t nat | wc -l
164757"><pre class="notranslate"><code class="notranslate"># sudo iptables -L -t nat | wc -l
164757
</code></pre></div>
<p dir="auto">The rules themselves:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# sudo iptables -L -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-PORTALS-CONTAINER  all  --  any    any     anywhere             anywhere            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */ 
    1    60 KUBE-PORTALS-CONTAINER  all  --  any    any     anywhere             anywhere            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */ 
    4   258 KUBE-PORTALS-CONTAINER  all  --  any    any     anywhere             anywhere            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */ 
    6   547 KUBE-PORTALS-CONTAINER  all  --  any    any     anywhere             anywhere            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */ 
   12  1632 KUBE-PORTALS-CONTAINER  all  --  any    any     anywhere             anywhere            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */ 
..."><pre class="notranslate"><code class="notranslate"># sudo iptables -L -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-PORTALS-CONTAINER  all  --  any    any     anywhere             anywhere            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */ 
    1    60 KUBE-PORTALS-CONTAINER  all  --  any    any     anywhere             anywhere            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */ 
    4   258 KUBE-PORTALS-CONTAINER  all  --  any    any     anywhere             anywhere            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */ 
    6   547 KUBE-PORTALS-CONTAINER  all  --  any    any     anywhere             anywhere            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */ 
   12  1632 KUBE-PORTALS-CONTAINER  all  --  any    any     anywhere             anywhere            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */ 
...
</code></pre></div>
<p dir="auto">The <code class="notranslate">OUTPUT</code> chain is just as bad as the <code class="notranslate">PREROUTING</code> chain. The <code class="notranslate">KUBE-*</code> chains are all sane, having one rule per service port.</p>
<p dir="auto">For now, I've set up a cron job to periodically flush <code class="notranslate">PREROUTING</code> and <code class="notranslate">OUTPUT</code> (and recreate the proper 4 rules to hopefully avoid any downtime), but this is obviously a poor workaround.</p>
<p dir="auto">I'm running release 1.0.6 on CentOS 6.</p>