<p dir="auto">I ran into an issue on our cluster running v1.0 with too many rules being specified for our node security group. Basically, security groups generated by Kubernetes were not cleaned up and their references in the node security group were also left behind. We hit an API limit (seems like it was ~50 rules defined on our node security group) which prevented any more load balancers from getting created.</p>
<p dir="auto">I started looking into this and it seems the issue still exists on master. From a cluster I built off of master (about a week ago) I was able to reproduce this issue.</p>
<p dir="auto">The controller manager logs look like this (filtered to aws.go and cleaned up):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="aws.go:1688] EnsureTCPLoadBalancer(a4981847bb57811e5b53d0ed29d18a59, us-east-1, \u003cnil\u003e, [0xc2083b5720], [])\n&quot;,&quot;stream&quot;:&quot;stderr&quot;,&quot;time&quot;:&quot;2016-01-13T21:58:00.802283892Z&quot;}
aws.go:1933] Removing rule for traffic from the load balancer (sg-294fcd50) to instance (sg-a1bb39d8)\n&quot;,&quot;stream&quot;:&quot;stderr&quot;,&quot;time&quot;:&quot;2016-01-13T21:58:01.348864017Z&quot;}
aws.go:1960] Revoking ingress was not needed; concurrent change? groupId=sg-a1bb39d8\n&quot;,&quot;stream&quot;:&quot;stderr&quot;,&quot;time&quot;:&quot;2016-01-13T21:58:01.412753316Z&quot;}
aws.go:1796] Loadbalancer a4981847bb57811e5b53d0ed29d18a59 has DNS name a4981847bb57811e5b53d0ed29d18a59-34625035.us-east-1.elb.amazonaws.com\n&quot;,&quot;stream&quot;:&quot;stderr&quot;,&quot;time&quot;:&quot;2016-01-13T21:58:01.487755376Z&quot;}
aws.go:1980] Load balancer already deleted: a492de23eb57811e5b53d0ed29d18a59\n&quot;,&quot;stream&quot;:&quot;stderr&quot;,&quot;time&quot;:&quot;2016-01-13T21:59:43.076849385Z&quot;}
aws.go:1933] Removing rule for traffic from the load balancer (sg-294fcd50) to instance (sg-a1bb39d8)\n&quot;,&quot;stream&quot;:&quot;stderr&quot;,&quot;time&quot;:&quot;2016-01-13T21:59:44.271066717Z&quot;}
aws.go:1960] Revoking ingress was not needed; concurrent change? groupId=sg-a1bb39d8\n&quot;,&quot;stream&quot;:&quot;stderr&quot;,&quot;time&quot;:&quot;2016-01-13T21:59:44.31932039Z&quot;}
aws.go:2034] Ignoring DependencyViolation while deleting load-balancer security group (sg-294fcd50), assuming because LB is in process of deleting\n&quot;,&quot;stream&quot;:&quot;stderr&quot;,&quot;time&quot;:&quot;2016-01-13T21:59:44.505570103Z&quot;}
aws.go:2053] Waiting for load-balancer to delete so we can delete security groups: a4981847bb57811e5b53d0ed29d18a59\n&quot;,&quot;stream&quot;:&quot;stderr&quot;,&quot;time&quot;:&quot;2016-01-13T21:59:44.505877497Z&quot;}"><pre class="notranslate"><code class="notranslate">aws.go:1688] EnsureTCPLoadBalancer(a4981847bb57811e5b53d0ed29d18a59, us-east-1, \u003cnil\u003e, [0xc2083b5720], [])\n","stream":"stderr","time":"2016-01-13T21:58:00.802283892Z"}
aws.go:1933] Removing rule for traffic from the load balancer (sg-294fcd50) to instance (sg-a1bb39d8)\n","stream":"stderr","time":"2016-01-13T21:58:01.348864017Z"}
aws.go:1960] Revoking ingress was not needed; concurrent change? groupId=sg-a1bb39d8\n","stream":"stderr","time":"2016-01-13T21:58:01.412753316Z"}
aws.go:1796] Loadbalancer a4981847bb57811e5b53d0ed29d18a59 has DNS name a4981847bb57811e5b53d0ed29d18a59-34625035.us-east-1.elb.amazonaws.com\n","stream":"stderr","time":"2016-01-13T21:58:01.487755376Z"}
aws.go:1980] Load balancer already deleted: a492de23eb57811e5b53d0ed29d18a59\n","stream":"stderr","time":"2016-01-13T21:59:43.076849385Z"}
aws.go:1933] Removing rule for traffic from the load balancer (sg-294fcd50) to instance (sg-a1bb39d8)\n","stream":"stderr","time":"2016-01-13T21:59:44.271066717Z"}
aws.go:1960] Revoking ingress was not needed; concurrent change? groupId=sg-a1bb39d8\n","stream":"stderr","time":"2016-01-13T21:59:44.31932039Z"}
aws.go:2034] Ignoring DependencyViolation while deleting load-balancer security group (sg-294fcd50), assuming because LB is in process of deleting\n","stream":"stderr","time":"2016-01-13T21:59:44.505570103Z"}
aws.go:2053] Waiting for load-balancer to delete so we can delete security groups: a4981847bb57811e5b53d0ed29d18a59\n","stream":"stderr","time":"2016-01-13T21:59:44.505877497Z"}
</code></pre></div>
<p dir="auto">The controller will loop until it hits the timeout and then bail out without having cleaned up the security group. I think the initial issue comes from <code class="notranslate">Revoking ingress was not needed; concurrent change? groupId=sg-a1bb39d8</code>. I verified through AWS console and that ingress rule was never removed. With that ingress rule still present a dependency on the security group still exists and the delete fails to happen (causing the dependency violation).</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/justinsb/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/justinsb">@justinsb</a> fyi.</p>
<p dir="auto">I'm looking into what I believe to be the culprit now and will push a PR if I find a solution.</p>
<p dir="auto">Edit:</p>
<p dir="auto">I have a strong suspicion that the culprit is <code class="notranslate">isEqualIPPermission</code>. It looks as if the AWS api is returning <code class="notranslate">IpPermission.IpRanges</code> while when we define the <code class="notranslate">IpPermission</code> we don't explicitly specify any <code class="notranslate">IpRanges</code>. It seems it would be reasonable to not do this check when removing ingress rules?</p>
<p dir="auto">ref <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="141756271" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/23174" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/23174/hovercard" href="https://github.com/kubernetes/kubernetes/issues/23174">#23174</a></p>