<p dir="auto">Process for bringing up an etcd cluster in a PetSet (minimal requirements).  Some discussion of races and inconsistencies.</p>
<ul dir="auto">
<li><a href="https://coreos.com/etcd/docs/latest/clustering.html" rel="nofollow">https://coreos.com/etcd/docs/latest/clustering.html</a></li>
<li><a href="https://coreos.com/etcd/docs/latest/runtime-configuration.html" rel="nofollow">https://coreos.com/etcd/docs/latest/runtime-configuration.html</a></li>
</ul>
<h4 dir="auto">Initialization (option1)</h4>
<ol dir="auto">
<li>Decide on a unique <code class="notranslate">--initial-cluster-token</code> and ensure that will be every pod as an ENV var (could be the petset UID or a simple parameter value)</li>
<li>Create all pods and unique PVs for each</li>
<li>Wait for all endpoints to be reachable to reflect all pods (requires we either know the quorum size, or know that the pet set has reached desired size).  Pods won't be ready, so we need to wait for ready+unready endpoints to == quorum size</li>
<li>etcd process starts with the initial bootstrap args (knowing what other members are in the set)</li>
</ol>
<p dir="auto">After this, no dynamic reconfiguration is allowed.</p>
<h4 dir="auto">Initialization (option2)</h4>
<ol dir="auto">
<li>Decide on a unique <code class="notranslate">--initial-cluster-token</code> and ensure that will be every pod as an ENV var (could be the petset UID or a simple parameter value)</li>
<li>Have the pod that gets identity "1" (or "master", or "first") create a new cluster</li>
<li>Have a control loop (babysitter / etc) running with pod identity 1 that tries to keep the cluster membership up to date with the endpoints list in etcd.</li>
</ol>
<p dir="auto">This allows dynamic reconfiguration, but is subject to duplicate instances of the babysitter running (discussed below).  It is possible to write the babysitter so that cluster membership always moves forward, but not possible to guarantee that two babysitters aren't running at the same time.</p>
<h4 dir="auto">Dealing with duplicate members without PV to act as a lock</h4>
<p dir="auto">It is possible that a kubelet becomes isolated from the cluster while running member 1 of a 3 pod set, and etcd is not configured to use PV (which acts as a form of lock in some cases).  The node controller detects that the kubelet is dead and triggers deletion of those pods.  After graceful deletion elapses, a new pod with member 3 will be created.  At this point, there are potentially two member-1's running.</p>
<p dir="auto">In option2 above, there could be two control loops running at the same time, and different pods could also see different endpoint versions (so the separated pod could see the older endpoint list).</p>
<p dir="auto">In option1 above, the kubelet could get isolated during initialization, and two instances of member-1 could be started.  Both would initialize, and the control loops in each would try to acquire members.  If sufficiently large numbers of nodes experienced partition, both sides could think they had a quorum.</p>
<p dir="auto">The resolution is to require the babysitters to make a write to the master API in a way that collapses one side or the other (such as writing to the endpoints).</p>
<h4 dir="auto">Upgrade (normal)</h4>
<ol dir="auto">
<li>Are all members healthy with a quorum?</li>
<li>Backup each member PV</li>
<li>Ensure clean shutdown of members on old version up to <code class="notranslate">floor(N / 2)</code>, to leave a quorum running</li>
<li>Start a new pod with the same PV as the old pod</li>
</ol>
<p dir="auto">During upgrade the same membership problems become possible as during initialization.  As long as the set has a quorum, it can police it's own membership, but a disruptive event during upgrade that disables quorum means that the babysitters would have to write to the master API in a similar way as during initialization.</p>
<h4 dir="auto">Upgrade (disaster)</h4>
<p dir="auto">In a disaster scenario, a majority of the PVs in the etcd cluster are lost.  No quorum is possible, but again a babysitter can rely on the external master API to resolve leadership for a period of time and reestablish quorum and ensure that the new members join the appropriate quorum.</p>
<h4 dir="auto">Thoughts</h4>
<p dir="auto">Even though etcd has its own consistent mechanism for managing membership, that requires a quorum.  Since the babysitter is itself potentially distributed, it must obtain a lease / lock to change membership in a strongly consistent fashion.  Only membership changes (pod creation / deletion, endpoint changes) require the babysitter to act in this fashion.  The remainder of the time, etcd can manage its own quorum state.</p>
<p dir="auto">If membership changes are relatively rare, there could be advantages to viewing the babysitter as a "cluster change hook".  The babysitter could be a run-once pod with a long duration grace period (lease, effectively) that is invoked on cluster membership changes, and the name of the pod could act as the lease (since writes to the master API are strongly consistent).  If the node running the babysitter goes down, it's possible to break the lock by deleting the pod forcefully with the corresponding lack of control over the outcome of any code still running in the pod.</p>
<p dir="auto">A one shot pod on membership changes has some advantages - limited resource consumption for the stability of the pod, easier potentially to reason about.</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto">No</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<p dir="auto">dnsmasq log</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):<br>
FEATURE REQUEST</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto">1.3.4</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: bare-metal</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): CoreOS 1068.8.0</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): Linux $REDACTED 4.6.3-coreos <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35192602" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/2" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/2/hovercard" href="https://github.com/kubernetes/kubernetes/issues/2">#2</a> SMP Mon Jul 18 06:10:39 UTC 2016 x86_64 Intel(R) Xeon(R) CPU E5530 @ 2.40GHz GenuineIntel GNU/Linux</li>
<li><strong>Install tools</strong>: bootcfg and ignition</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<ol dir="auto">
<li>I started kube-dns according to this example: <a href="https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/skydns-rc.yaml.base">https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/skydns-rc.yaml.base</a></li>
<li><code class="notranslate">kubectl logs --tail=100 -f kube-dns-v19-????? dnsmasq</code> did not show anything.</li>
</ol>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto"><code class="notranslate">kubectl logs --tail=100 -f kube-dns-v19-????? dnsmasq</code> should have shown log messages. There should have been at least a start-up message and it should possibly also log DNS requests, as does the <code class="notranslate">kubedns</code> container.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">Take the example <code class="notranslate">kube-dns-rc.yaml</code> and create a resource controller from it.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">For my DHCP/TFTP dnsmasq pod these command line switches have the desired effect: <code class="notranslate">--keep-in-foreground --log-facility=-</code></p>