<p dir="auto"><a href="https://github.com/kubernetes/kubernetes/blob/v1.6.0-alpha.0/pkg/util/strategicpatch/patch.go#L659">StrategicPatch</a> should delete all duplicates from a list of maps.<br>
Currently it just delete the first one appears in the list.</p>
<p dir="auto">We should make StrategicPatch delete all duplicates no matter it is a list of maps or a list of primitives. (StrategicPatch don't support list of lists).</p>
<p dir="auto">Ref : <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="193254157" data-permission-text="Title is private" data-url="https://github.com/kubernetes/community/issues/140" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/community/pull/140/hovercard?comment_id=265598220&amp;comment_type=issue_comment" href="https://github.com/kubernetes/community/pull/140#issuecomment-265598220">kubernetes/community#140 (comment)</a></p>
<p dir="auto">cc: <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/lavalamp/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/lavalamp">@lavalamp</a></p>
<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):<br>
Upgrading from 1.2.0 (working) to 1.3.9 (broken)</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: Bare metal Dell R420's</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): CentOS 6 with a 3.10 kernel.</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):  3.10.0-327.18.2.el7.x86_64</li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:<br>
iptables --version<br>
iptables v1.4.7</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
We upgraded from 1.2.0 to 1.3.9 and noticed that our service load balancer (based on haproxy) cpu utilization started going through the roof over the course of 3 days, mostly system time.  It is now impacting all nodes with the 1.3.9 version of kubelet installed.</p>
<p dir="auto">We've tracked down a likely culprit which is this version of kube seems to be continuously appending iptables rules at a rate of about 4 a second.  Here's a dump from iptables-save:<br>
$ sudo iptables-save | wc -l<br>
67951</p>
<p dir="auto">These tend to be the form of the duplicated rules:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="-A KUBE-FIREWALL -m comment --comment &quot;kubernetes firewall for dropping marked packets&quot; -m mark --mark 0x8000/0x8000 -j DROP
&lt;repeated&gt;
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
&lt;repeated&gt;
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
&lt;repeated&gt;
-A KUBE-POSTROUTING -m comment --comment &quot;kubernetes service traffic requiring SNAT&quot; -m mark --mark 0x4000/0x4000 -j MASQUERADE
&lt;repeated&gt;"><pre class="notranslate"><code class="notranslate">-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
&lt;repeated&gt;
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
&lt;repeated&gt;
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
&lt;repeated&gt;
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
&lt;repeated&gt;
</code></pre></div>
<p dir="auto">The commits and issues I've found related to this are:<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="50146259" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/2623" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/2623/hovercard" href="https://github.com/kubernetes/kubernetes/issues/2623">#2623</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="170774439" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/30486" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/30486/hovercard" href="https://github.com/kubernetes/kubernetes/pull/30486">#30486</a></p>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
It to stay at &lt; 30% cpu utilization and a reasonable number of iptables rules.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):<br>
Setup centos6 with our kernel, run 1.3.9</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">Here is the iptables output soon after starting the system.  You can see it is already starting to accumulate rules:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ sudo iptables-save
# Generated by iptables-save v1.4.7 on Wed Nov 23 16:35:34 2016
*filter
:INPUT ACCEPT [2753:1417441]
:FORWARD ACCEPT [107:7101]
:OUTPUT ACCEPT [1993:8997998]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:KUBE-FIREWALL - [0:0]
-A INPUT -j KUBE-FIREWALL
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
-A KUBE-FIREWALL -m comment --comment &quot;kubernetes firewall for dropping marked packets&quot; -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL -m comment --comment &quot;kubernetes firewall for dropping marked packets&quot; -m mark --mark 0x8000/0x8000 -j DROP
COMMIT
# Completed on Wed Nov 23 16:35:34 2016
# Generated by iptables-save v1.4.7 on Wed Nov 23 16:35:34 2016
*nat
:PREROUTING ACCEPT [462:40180]
:INPUT ACCEPT [136:16306]
:OUTPUT ACCEPT [1007:65897]
:POSTROUTING ACCEPT [1109:72017]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-POSTROUTING - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A PREROUTING ! -s 127.0.0.1/32 -p tcp -m tcp --dport 10250 -m addrtype --dst-type LOCAL -j REDIRECT --to-ports 11250
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment &quot;kubernetes postrouting rules&quot; -j KUBE-POSTROUTING
-A POSTROUTING -s 10.20.19.128/25 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment &quot;kubernetes service traffic requiring SNAT&quot; -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-POSTROUTING -m comment --comment &quot;kubernetes service traffic requiring SNAT&quot; -m mark --mark 0x4000/0x4000 -j MASQUERADE
COMMIT
# Completed on Wed Nov 23 16:35:34 2016"><pre class="notranslate"><code class="notranslate">$ sudo iptables-save
# Generated by iptables-save v1.4.7 on Wed Nov 23 16:35:34 2016
*filter
:INPUT ACCEPT [2753:1417441]
:FORWARD ACCEPT [107:7101]
:OUTPUT ACCEPT [1993:8997998]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:KUBE-FIREWALL - [0:0]
-A INPUT -j KUBE-FIREWALL
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
COMMIT
# Completed on Wed Nov 23 16:35:34 2016
# Generated by iptables-save v1.4.7 on Wed Nov 23 16:35:34 2016
*nat
:PREROUTING ACCEPT [462:40180]
:INPUT ACCEPT [136:16306]
:OUTPUT ACCEPT [1007:65897]
:POSTROUTING ACCEPT [1109:72017]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-POSTROUTING - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A PREROUTING ! -s 127.0.0.1/32 -p tcp -m tcp --dport 10250 -m addrtype --dst-type LOCAL -j REDIRECT --to-ports 11250
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 10.20.19.128/25 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
COMMIT
# Completed on Wed Nov 23 16:35:34 2016
</code></pre></div>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/thockin/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/thockin">@thockin</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/freehan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/freehan">@freehan</a></p>