<p dir="auto">I'm working on standing up a 3 node Zookeeper instance, and instead of having separate RCs and Services for each individual pod, I'd like one RC with three instances that keeps all three pods alive. The image is based off the <code class="notranslate">mesoscloud/zookeeper</code> image, just with the quorum enabled.</p>
<p dir="auto">zookeeper-rc.json</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    &quot;apiVersion&quot;: &quot;v1&quot;,
    &quot;kind&quot;: &quot;ReplicationController&quot;,
    &quot;metadata&quot;: {
        &quot;name&quot;: &quot;zookeeper-test&quot;
    },
    &quot;spec&quot;: {
        &quot;replicas&quot;: 1,
        &quot;template&quot;: {
            &quot;metadata&quot;: {
                &quot;labels&quot;: {
                    &quot;app&quot;: &quot;zookeeper&quot;
                }
            },
            &quot;spec&quot;: {
                &quot;containers&quot;: [
                    {
                        &quot;env&quot;: [
                            {
                                &quot;name&quot;: &quot;MYID&quot;,
                                &quot;value&quot;: &quot;3&quot;
                            },
                            {
                                &quot;name&quot;: &quot;SERVERS&quot;,
                                &quot;value&quot;: &quot;zk1,zk2,zk3&quot;
                            }
                        ],
                        &quot;image&quot;: &quot;custom/zookeeper:latest&quot;,
                        &quot;name&quot;: &quot;zk3&quot;,
                        &quot;ports&quot;: [
                            {
                                &quot;containerPort&quot;: 2181
                            },
                            {
                                &quot;containerPort&quot;: 2888
                            },
                            {
                                &quot;containerPort&quot;: 3888
                            }
                        ]
                    },
                    {
                        &quot;env&quot;: [
                            {
                                &quot;name&quot;: &quot;MYID&quot;,
                                &quot;value&quot;: &quot;2&quot;
                            },
                            {
                                &quot;name&quot;: &quot;SERVERS&quot;,
                                &quot;value&quot;: &quot;zk1,zk2,zk3&quot;
                            }
                        ],
                        &quot;image&quot;: &quot;custom/zookeeper:latest&quot;,
                        &quot;name&quot;: &quot;zk2&quot;,
                        &quot;ports&quot;: [
                            {
                                &quot;containerPort&quot;: 2181
                            },
                            {
                                &quot;containerPort&quot;: 2888
                            },
                            {
                                &quot;containerPort&quot;: 3888
                            }
                        ]
                    },
                    {
                        &quot;env&quot;: [
                            {
                                &quot;name&quot;: &quot;MYID&quot;,
                                &quot;value&quot;: &quot;1&quot;
                            },
                            {
                                &quot;name&quot;: &quot;SERVERS&quot;,
                                &quot;value&quot;: &quot;zk1,zk2,zk3&quot;
                            }
                        ],
                        &quot;image&quot;: &quot;custom/zookeeper:latest&quot;,
                        &quot;name&quot;: &quot;zk1&quot;,
                        &quot;ports&quot;: [
                            {
                                &quot;containerPort&quot;: 2181
                            },
                            {
                                &quot;containerPort&quot;: 2888
                            },
                            {
                                &quot;containerPort&quot;: 3888
                            }
                        ]
                    }
                ]
            }
        }
    }
}"><pre class="notranslate">{
    <span class="pl-ent">"apiVersion"</span>: <span class="pl-s"><span class="pl-pds">"</span>v1<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"kind"</span>: <span class="pl-s"><span class="pl-pds">"</span>ReplicationController<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"metadata"</span>: {
        <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>zookeeper-test<span class="pl-pds">"</span></span>
    },
    <span class="pl-ent">"spec"</span>: {
        <span class="pl-ent">"replicas"</span>: <span class="pl-c1">1</span>,
        <span class="pl-ent">"template"</span>: {
            <span class="pl-ent">"metadata"</span>: {
                <span class="pl-ent">"labels"</span>: {
                    <span class="pl-ent">"app"</span>: <span class="pl-s"><span class="pl-pds">"</span>zookeeper<span class="pl-pds">"</span></span>
                }
            },
            <span class="pl-ent">"spec"</span>: {
                <span class="pl-ent">"containers"</span>: [
                    {
                        <span class="pl-ent">"env"</span>: [
                            {
                                <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>MYID<span class="pl-pds">"</span></span>,
                                <span class="pl-ent">"value"</span>: <span class="pl-s"><span class="pl-pds">"</span>3<span class="pl-pds">"</span></span>
                            },
                            {
                                <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>SERVERS<span class="pl-pds">"</span></span>,
                                <span class="pl-ent">"value"</span>: <span class="pl-s"><span class="pl-pds">"</span>zk1,zk2,zk3<span class="pl-pds">"</span></span>
                            }
                        ],
                        <span class="pl-ent">"image"</span>: <span class="pl-s"><span class="pl-pds">"</span>custom/zookeeper:latest<span class="pl-pds">"</span></span>,
                        <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>zk3<span class="pl-pds">"</span></span>,
                        <span class="pl-ent">"ports"</span>: [
                            {
                                <span class="pl-ent">"containerPort"</span>: <span class="pl-c1">2181</span>
                            },
                            {
                                <span class="pl-ent">"containerPort"</span>: <span class="pl-c1">2888</span>
                            },
                            {
                                <span class="pl-ent">"containerPort"</span>: <span class="pl-c1">3888</span>
                            }
                        ]
                    },
                    {
                        <span class="pl-ent">"env"</span>: [
                            {
                                <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>MYID<span class="pl-pds">"</span></span>,
                                <span class="pl-ent">"value"</span>: <span class="pl-s"><span class="pl-pds">"</span>2<span class="pl-pds">"</span></span>
                            },
                            {
                                <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>SERVERS<span class="pl-pds">"</span></span>,
                                <span class="pl-ent">"value"</span>: <span class="pl-s"><span class="pl-pds">"</span>zk1,zk2,zk3<span class="pl-pds">"</span></span>
                            }
                        ],
                        <span class="pl-ent">"image"</span>: <span class="pl-s"><span class="pl-pds">"</span>custom/zookeeper:latest<span class="pl-pds">"</span></span>,
                        <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>zk2<span class="pl-pds">"</span></span>,
                        <span class="pl-ent">"ports"</span>: [
                            {
                                <span class="pl-ent">"containerPort"</span>: <span class="pl-c1">2181</span>
                            },
                            {
                                <span class="pl-ent">"containerPort"</span>: <span class="pl-c1">2888</span>
                            },
                            {
                                <span class="pl-ent">"containerPort"</span>: <span class="pl-c1">3888</span>
                            }
                        ]
                    },
                    {
                        <span class="pl-ent">"env"</span>: [
                            {
                                <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>MYID<span class="pl-pds">"</span></span>,
                                <span class="pl-ent">"value"</span>: <span class="pl-s"><span class="pl-pds">"</span>1<span class="pl-pds">"</span></span>
                            },
                            {
                                <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>SERVERS<span class="pl-pds">"</span></span>,
                                <span class="pl-ent">"value"</span>: <span class="pl-s"><span class="pl-pds">"</span>zk1,zk2,zk3<span class="pl-pds">"</span></span>
                            }
                        ],
                        <span class="pl-ent">"image"</span>: <span class="pl-s"><span class="pl-pds">"</span>custom/zookeeper:latest<span class="pl-pds">"</span></span>,
                        <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>zk1<span class="pl-pds">"</span></span>,
                        <span class="pl-ent">"ports"</span>: [
                            {
                                <span class="pl-ent">"containerPort"</span>: <span class="pl-c1">2181</span>
                            },
                            {
                                <span class="pl-ent">"containerPort"</span>: <span class="pl-c1">2888</span>
                            },
                            {
                                <span class="pl-ent">"containerPort"</span>: <span class="pl-c1">3888</span>
                            }
                        ]
                    }
                ]
            }
        }
    }
}</pre></div>
<p dir="auto">I have them working with one container per RC, but then I have one RC for each pod. Right now this is just testing, but when I scale out to 7+ nodes, and occassionally 13 nodes, I don't want 13 RCs for one ZK deployment.</p>
<p dir="auto">When I try to deploy this:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="mynameismevin$ kubectl create -f zookeeper-rc.json --validate
The ReplicationController &quot;zookeeper-rc&quot; is invalid:
* spec.template.spec.containers[1].name: duplicate value 'server'
* spec.template.spec.containers[2].name: duplicate value 'server'"><pre class="notranslate">mynameismevin$ kubectl create -f zookeeper-rc.json --validate
The ReplicationController <span class="pl-s"><span class="pl-pds">"</span>zookeeper-rc<span class="pl-pds">"</span></span> is invalid:
<span class="pl-k">*</span> spec.template.spec.containers[1].name: duplicate value <span class="pl-s"><span class="pl-pds">'</span>server<span class="pl-pds">'</span></span>
<span class="pl-k">*</span> spec.template.spec.containers[2].name: duplicate value <span class="pl-s"><span class="pl-pds">'</span>server<span class="pl-pds">'</span></span></pre></div>
<p dir="auto">The EnvVars have to have <code class="notranslate">"name":"server"</code> for each container, so that can be passed to ZK so it knows what servers are in the cluster.</p>
<p dir="auto">Is this a limitation of k8s or am I using this incorrectly?</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):<br>
No</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):<br>
<code class="notranslate">scheduledjob</code>, <code class="notranslate">cannot determine</code>, <code class="notranslate">too many missed start times</code></p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):<br>
BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;4&quot;, GitVersion:&quot;v1.4.0+coreos.0&quot;, GitCommit:&quot;278a1f7034bdba61cba443722647da1a8204a6fc&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-09-26T20:48:37Z&quot;, GoVersion:&quot;go1.6.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;4&quot;, GitVersion:&quot;v1.4.0+coreos.0&quot;, GitCommit:&quot;278a1f7034bdba61cba443722647da1a8204a6fc&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-09-26T20:48:37Z&quot;, GoVersion:&quot;go1.6.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0+coreos.0", GitCommit:"278a1f7034bdba61cba443722647da1a8204a6fc", GitTreeState:"clean", BuildDate:"2016-09-26T20:48:37Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0+coreos.0", GitCommit:"278a1f7034bdba61cba443722647da1a8204a6fc", GitTreeState:"clean", BuildDate:"2016-09-26T20:48:37Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: bare metal</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): <code class="notranslate">Debian GNU/Linux 8 (jessie)</code></li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): <code class="notranslate">4.4.0-36-generic #55-Ubuntu SMP Thu Aug 11 18:01:55 UTC 2016 x86_64 GNU/Linux</code></li>
<li><strong>Install tools</strong>: <code class="notranslate">quay.io/coreos/hyperkube:v1.4.0_coreos.0</code>, deployed with <code class="notranslate">kargo</code></li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">I am using a scheduledjob with <strong><code class="notranslate">concurrencyPolicy: Forbid</code></strong>. An instance of my job has taken a long time to finish, thus skipping many start times.</p>
<p dir="auto">Ever since, the scheduledjob will not schedule anything new.</p>
<p dir="auto">Controller manager log has:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="E1106 13:37:12.457445       1 controller.go:163] Cannot determine if default/my-cron-job needs to be started: Too many missed start times to list"><pre class="notranslate"><code class="notranslate">E1106 13:37:12.457445       1 controller.go:163] Cannot determine if default/my-cron-job needs to be started: Too many missed start times to list
</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">I expect the scheduledjob not to be stuck not scheduling anything anymore. With <code class="notranslate">concurrencyPolicy: Forbid</code>, I expect that it should just re-schedule something as soon as the previously running job has completed.</p>
<p dir="auto">The "Too many missed start times to list" safeguard appears to apply mostly to cases where job concurrency is allowed to avoid creating a lot of jobs in case of timestamp issues.<br>
</p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/kubernetes/kubernetes/blob/56bbfd259a6f45dfbcfb6f3c9f61643a787ed067/pkg/controller/scheduledjob/utils.go#L148">kubernetes/pkg/controller/scheduledjob/utils.go</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 148
      in
      <a data-pjax="true" class="commit-tease-sha" href="/kubernetes/kubernetes/commit/56bbfd259a6f45dfbcfb6f3c9f61643a787ed067">56bbfd2</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L148" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="148"></td>
          <td id="LC148" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-c">// An object might miss several starts.  For example, if</span> </td>
        </tr>
    </tbody></table>
  </div>
</div>
:<p></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="		// An object might miss several starts.  For example, if
		// controller gets wedged on friday at 5:01pm when everyone has
		// gone home, and someone comes in on tuesday AM and discovers
		// the problem and restarts the controller, then all the hourly
		// jobs, more than 80 of them for one hourly scheduledJob, should
		// all start running with no further intervention (if the scheduledJob
		// allows concurrency and late starts).
		//

		// However, if there is a bug somewhere, or incorrect clock
		// on controller's server or apiservers (for setting creationTimestamp)
		// then there could be so many missed start times (it could be off
		// by decades or more), that it would eat up all the CPU and memory
		// of this controller. In that case, we want to not try to list
		// all the misseded start times."><pre class="notranslate"><code class="notranslate">		// An object might miss several starts.  For example, if
		// controller gets wedged on friday at 5:01pm when everyone has
		// gone home, and someone comes in on tuesday AM and discovers
		// the problem and restarts the controller, then all the hourly
		// jobs, more than 80 of them for one hourly scheduledJob, should
		// all start running with no further intervention (if the scheduledJob
		// allows concurrency and late starts).
		//

		// However, if there is a bug somewhere, or incorrect clock
		// on controller's server or apiservers (for setting creationTimestamp)
		// then there could be so many missed start times (it could be off
		// by decades or more), that it would eat up all the CPU and memory
		// of this controller. In that case, we want to not try to list
		// all the misseded start times.
</code></pre></div>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">The issue shows up after about 100 minutes of this running.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apiVersion: batch/v2alpha1
kind: ScheduledJob
metadata:
  name: my-cron-job
  labels:
    app: my-cron-job
spec:
  schedule: 0/1 * * * ?
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: my-cron-job
            image: busybox:1.25
            command: [&quot;sleep&quot;, &quot;7000&quot;]
          restartPolicy: Never"><pre class="notranslate"><code class="notranslate">apiVersion: batch/v2alpha1
kind: ScheduledJob
metadata:
  name: my-cron-job
  labels:
    app: my-cron-job
spec:
  schedule: 0/1 * * * ?
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: my-cron-job
            image: busybox:1.25
            command: ["sleep", "7000"]
          restartPolicy: Never
</code></pre></div>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>