<p dir="auto">we have setup one kubernetes cluster to test the HA if support to transfer the container with the cinder volume plugin. I have three nodes. One Master Node, and two Nodes are minion nodes. I install with cinder volume plugin successfully.<br>
1.I use the case to create the pod on A minion node. It can run well. And the cinder can also be attached to the container.<br>
apiVersion: v1<br>
kind: ReplicationController<br>
metadata:<br>
name: wqonlynginx<br>
spec:<br>
replicas: 1<br>
selector:<br>
name: wqonlynginx<br>
template:<br>
metadata:<br>
labels:<br>
name: wqonlynginx<br>
spec:<br>
containers:<br>
- name: wqonlynginx<br>
image: nginx<br>
ports:<br>
- containerPort: 6379<br>
resources:<br>
limits:<br>
cpu: "0.1"<br>
volumeMounts:<br>
- mountPath: /nginx-data<br>
name: data<br>
volumes:<br>
- name: data<br>
cinder:<br>
# Enter the volume ID below<br>
volumeID: 37ef6468-dfde-4468-8317-2c4142a6827f<br>
fsType: ext4<br>
2. I stop the A minion by the command "nova stop instance_id", But I find the status of pod is ContainerCreating all the time. I find the cinder volume isn't attached to the B minion node if HA supports, and the B minion node don't find the container to be transfered.</p>
<p dir="auto">From our test, it seems kubernetes HA Not support this container with cinder volume plugin to transfer to the another minion node.<br>
Could any kubernetes developers confirmed that ?</p>
<p dir="auto">Even after extending/randomizing the server timeout I see too many watch window expirations. This leads to concurrent lists (it has dropped from &gt; 50, to &gt;30 &lt; 50). There are periods when we get one expiration a second, and times when we have &gt; 20 on the same second.</p>
<p dir="auto">I can boil this down to cpu starvation of the go-etcd watcher:<br>
<a href="https://github.com/coreos/go-etcd/blob/master/etcd/watch.go#L39">https://github.com/coreos/go-etcd/blob/master/etcd/watch.go#L39</a></p>
<p dir="auto">And reproduce it with:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="go test -bench BenchmarkPodList -bench-pods 3000 -bench-tasks 100 -bench-quiet 3 -tags 'benchmark'"><pre class="notranslate"><code class="notranslate">go test -bench BenchmarkPodList -bench-pods 3000 -bench-tasks 100 -bench-quiet 3 -tags 'benchmark'
</code></pre></div>
<p dir="auto">while concurrently setting up a cpu starved watch:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="func TestEtcdWatch() {
    client := etcd.NewClient()
    var yield sync.Mutex
    go func() {
        for {
            yield.Lock()
            yield.Unlock()
        }
    }()
    go func() {
        if _, err := client.Watch(Key, waitIndex, true, incomingCh, stopCh); err != nil {
            errCh &lt;- err
        }
    }()
    for {
        select {
        case res := &lt;-incomingCh:
            glog.Infof(&quot;Got result %v&quot;, res)
        case err := &lt;-errCh:
            glog.Fatalf(&quot;Got error %v&quot;, err)
        }
    }
}
"><pre class="notranslate"><code class="notranslate">func TestEtcdWatch() {
    client := etcd.NewClient()
    var yield sync.Mutex
    go func() {
        for {
            yield.Lock()
            yield.Unlock()
        }
    }()
    go func() {
        if _, err := client.Watch(Key, waitIndex, true, incomingCh, stopCh); err != nil {
            errCh &lt;- err
        }
    }()
    for {
        select {
        case res := &lt;-incomingCh:
            glog.Infof("Got result %v", res)
        case err := &lt;-errCh:
            glog.Fatalf("Got error %v", err)
        }
    }
}

</code></pre></div>
<p dir="auto">I don't think this is s pathological test case. We hammer cpu on our n1-standard-4 master, and in reality we have 100's of open watches (and each of those 3000 pods will send at least 7 events).</p>
<p dir="auto">There are a few solutions:</p>
<ol dir="auto">
<li>Make the etcd watch window a command line parameter and bump it up to 5000.</li>
<li>Etcd server supports stream watching.</li>
<li>We Implement a cache with a custom watch window.</li>
</ol>
<p dir="auto">I've tested 1 and it seems to work (haven't investigated tradeoffs of larger window). 2, 3 are probably not going to happen for 1.0.</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/lavalamp/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/lavalamp">@lavalamp</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/fgrzadkowski/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/fgrzadkowski">@fgrzadkowski</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/wojtek-t/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/wojtek-t">@wojtek-t</a></p>