<p dir="auto">Do we intend to support prior versions of APIs?  E.g. Should we<br>
support v1beta1 inputs once we support v1beta2?  To do that properly<br>
we need to version the whole API, not just individual structures, and<br>
we probably want version-specific code that translates old inputs to<br>
"most modern" form well isolated.  And Then do we support old versions<br>
on output?</p>
<p dir="auto">To do this "right" is very intensive.</p>
<p dir="auto">We also need to write up a policy statement regarding API changes and deprecation policies.</p>
<p dir="auto">BUG REPORT<br>
<strong>Kubernetes version</strong><br>
v1.4.3+coreos.0</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>:<br>
Bare metal CoreOS installation. 3 management nodes + 2 worker nodes.</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):<br>
<code class="notranslate">CoreOS stable (1185.3.0)</code></li>
<li><strong>Kernel</strong><br>
<code class="notranslate">4.7.3-coreos-r2</code></li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
DaemonSet pods randomly fail to start on management nodes after nodes reboot.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt; kubectl get pods -o wide -a -l app=ds-gluster-slow-01
NAME                    READY     STATUS              RESTARTS   AGE       IP            NODE
gluster-slow-01-fgpk3   1/1       Running             0          19h       10.54.147.5   10.54.147.5
gluster-slow-01-payvs   0/1       MatchNodeSelector   0          10h       &lt;none&gt;        10.54.147.6
gluster-slow-01-zdr0t   1/1       Running             0          19h       10.54.147.4   10.54.147.4"><pre class="notranslate"><code class="notranslate">&gt; kubectl get pods -o wide -a -l app=ds-gluster-slow-01
NAME                    READY     STATUS              RESTARTS   AGE       IP            NODE
gluster-slow-01-fgpk3   1/1       Running             0          19h       10.54.147.5   10.54.147.5
gluster-slow-01-payvs   0/1       MatchNodeSelector   0          10h       &lt;none&gt;        10.54.147.6
gluster-slow-01-zdr0t   1/1       Running             0          19h       10.54.147.4   10.54.147.4
</code></pre></div>
<p dir="auto">10.54.147.4, 10.54.147.5, 10.54.147.6 are the management nodes.<br>
The only way to start the pod again is to delete the pod. The <code class="notranslate">MatchNodeSelector</code> happens once every 2-3 reboots. For remaining reboots the pods are started properly.</p>
<p dir="auto"><strong>How to reproduce it</strong>:<br>
Create a DaemonSet and assign it to management nodes based on user defined labels.<br>
Here is our configuration:</p>
<p dir="auto">Hosts labels:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="NAME           STATUS                     AGE       LABELS
10.54.147.10   Ready                      52d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10.54.147.10,type=infra
10.54.147.11   Ready                      52d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10.54.147.11,type=infra
10.54.147.4    Ready                      53d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gluster=slow-01,kubernetes.io/hostname=10.54.147.4,type=management
10.54.147.5    Ready                      53d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gluster=slow-01,kubernetes.io/hostname=10.54.147.5,type=management
10.54.147.6    Ready,SchedulingDisabled   47d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gluster=slow-01,kubernetes.io/hostname=10.54.147.6,type=management"><pre class="notranslate"><code class="notranslate">NAME           STATUS                     AGE       LABELS
10.54.147.10   Ready                      52d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10.54.147.10,type=infra
10.54.147.11   Ready                      52d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10.54.147.11,type=infra
10.54.147.4    Ready                      53d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gluster=slow-01,kubernetes.io/hostname=10.54.147.4,type=management
10.54.147.5    Ready                      53d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gluster=slow-01,kubernetes.io/hostname=10.54.147.5,type=management
10.54.147.6    Ready,SchedulingDisabled   47d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gluster=slow-01,kubernetes.io/hostname=10.54.147.6,type=management
</code></pre></div>
<p dir="auto">DaemonSet definition:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    name: gluster-slow-01
  name: gluster-slow-01
  namespace: default
spec:
  selector:
    matchLabels:
      app: ds-gluster-slow-01
  template:
    metadata:
      labels:
        app: ds-gluster-slow-01
    spec:
      containers:
      - image: internal-registry/mic/infra/glusterfs:1
        name: gluster-slow-01
        livenessProbe:
          tcpSocket:
            port: 24007
          initialDelaySeconds: 30
          timeoutSeconds: 1
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /mnt/localdisk
          name: gluster-localdisk
        - mountPath: /var/lib/glusterd
          name: gluster-varlib
        - mountPath: /dev
          name: gluster-dev
        - mountPath: /sys/fs/cgroup
          name: gluster-cgroup
      dnsPolicy: ClusterFirst
      hostNetwork: true
      nodeSelector:
        gluster: slow-01
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 1
      volumes:
      - hostPath:
          path: /mnt/localdisk
        name: gluster-localdisk
      - hostPath:
          path: /dev
        name: gluster-dev
      - hostPath:
          path: /sys/fs/cgroup
        name: gluster-cgroup
      - hostPath:
          path: /var/lib/glusterd
        name: gluster-varlib"><pre class="notranslate"><code class="notranslate">apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    name: gluster-slow-01
  name: gluster-slow-01
  namespace: default
spec:
  selector:
    matchLabels:
      app: ds-gluster-slow-01
  template:
    metadata:
      labels:
        app: ds-gluster-slow-01
    spec:
      containers:
      - image: internal-registry/mic/infra/glusterfs:1
        name: gluster-slow-01
        livenessProbe:
          tcpSocket:
            port: 24007
          initialDelaySeconds: 30
          timeoutSeconds: 1
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /mnt/localdisk
          name: gluster-localdisk
        - mountPath: /var/lib/glusterd
          name: gluster-varlib
        - mountPath: /dev
          name: gluster-dev
        - mountPath: /sys/fs/cgroup
          name: gluster-cgroup
      dnsPolicy: ClusterFirst
      hostNetwork: true
      nodeSelector:
        gluster: slow-01
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 1
      volumes:
      - hostPath:
          path: /mnt/localdisk
        name: gluster-localdisk
      - hostPath:
          path: /dev
        name: gluster-dev
      - hostPath:
          path: /sys/fs/cgroup
        name: gluster-cgroup
      - hostPath:
          path: /var/lib/glusterd
        name: gluster-varlib
</code></pre></div>
<p dir="auto"><strong>Anything else do we need to know</strong>:<br>
The kube-scheduler yaml file for managment nodes is defined as follows:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apiVersion: v1
kind: Pod
metadata:
  name: kube-scheduler
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - name: kube-scheduler
    image: quay.io/coreos/hyperkube:v1.4.3_coreos.0
    command:
    - /hyperkube
    - scheduler
    - --master=http://127.0.0.1:8080
    - --leader-elect=true
    livenessProbe:
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10251
      initialDelaySeconds: 15
      timeoutSeconds: 1"><pre class="notranslate"><code class="notranslate">apiVersion: v1
kind: Pod
metadata:
  name: kube-scheduler
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - name: kube-scheduler
    image: quay.io/coreos/hyperkube:v1.4.3_coreos.0
    command:
    - /hyperkube
    - scheduler
    - --master=http://127.0.0.1:8080
    - --leader-elect=true
    livenessProbe:
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10251
      initialDelaySeconds: 15
      timeoutSeconds: 1
</code></pre></div>
<p dir="auto">The top log lines from the kube-scheduler pod are:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="E1109 06:51:42.487688       1 leaderelection.go:252] error retrieving endpoint: Get http://127.0.0.1:8080/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.662848       1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:394: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.662918       1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:404: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.662958       1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:391: Failed to list *api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&amp;resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.663004       1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:388: Failed to list *api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&amp;resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.663124       1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:414: Failed to list *extensions.ReplicaSet: Get http://127.0.0.1:8080/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.663170       1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:409: Failed to list *api.ReplicationController: Get http://127.0.0.1:8080/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.663212       1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:399: Failed to list *api.PersistentVolumeClaim: Get http://127.0.0.1:8080/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.663257       1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:398: Failed to list *api.PersistentVolume: Get http://127.0.0.1:8080/api/v1/persistentvolumes?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
I1109 06:51:45.940628       1 leaderelection.go:295] lock is held by sid-kb-004 and has not yet expired
I1109 06:51:48.664098       1 leaderelection.go:295] lock is held by sid-kb-004 and has not yet expired"><pre class="notranslate"><code class="notranslate">E1109 06:51:42.487688       1 leaderelection.go:252] error retrieving endpoint: Get http://127.0.0.1:8080/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.662848       1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:394: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.662918       1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:404: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.662958       1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:391: Failed to list *api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&amp;resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.663004       1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:388: Failed to list *api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&amp;resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.663124       1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:414: Failed to list *extensions.ReplicaSet: Get http://127.0.0.1:8080/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.663170       1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:409: Failed to list *api.ReplicationController: Get http://127.0.0.1:8080/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.663212       1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:399: Failed to list *api.PersistentVolumeClaim: Get http://127.0.0.1:8080/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.663257       1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:398: Failed to list *api.PersistentVolume: Get http://127.0.0.1:8080/api/v1/persistentvolumes?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
I1109 06:51:45.940628       1 leaderelection.go:295] lock is held by sid-kb-004 and has not yet expired
I1109 06:51:48.664098       1 leaderelection.go:295] lock is held by sid-kb-004 and has not yet expired
</code></pre></div>
<p dir="auto">I suspect it is a race condition during the management pods initialization. The <code class="notranslate">kube-scheduler</code> is started before the <code class="notranslate">kube-apiserver</code> pod. It may not be handling well the fact that it can't connect to the Kubernetes api service.<br>
I tried to workaround it by adding the <code class="notranslate">init-containers</code> section:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apiVersion: v1
kind: Pod
metadata:
  name: kube-scheduler
  namespace: kube-system
  annotations:
    pod.beta.kubernetes.io/init-containers: '[
      {
        &quot;image&quot;: &quot;quay.io/coreos/hyperkube:v1.4.3_coreos.0&quot;,
        &quot;name&quot;: &quot;wait-for-master&quot;,
        &quot;command&quot;: [ &quot;/bin/bash&quot;, &quot;-c&quot;, &quot;while ! timeout 1 bash -c \&quot;/kubectl --server=http://127.0.0.1:8080/ cluster-info\&quot;; do sleep 1; done&quot; ]
      }
    ]'
spec:
  hostNetwork: true
  containers:
  - name: kube-scheduler
    image: quay.io/coreos/hyperkube:v1.4.3_coreos.0
    command:
    - /hyperkube
    - scheduler
    - --master=http://127.0.0.1:8080
    - --leader-elect=true
    livenessProbe:
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10251
      initialDelaySeconds: 15
      timeoutSeconds: 1"><pre class="notranslate"><code class="notranslate">apiVersion: v1
kind: Pod
metadata:
  name: kube-scheduler
  namespace: kube-system
  annotations:
    pod.beta.kubernetes.io/init-containers: '[
      {
        "image": "quay.io/coreos/hyperkube:v1.4.3_coreos.0",
        "name": "wait-for-master",
        "command": [ "/bin/bash", "-c", "while ! timeout 1 bash -c \"/kubectl --server=http://127.0.0.1:8080/ cluster-info\"; do sleep 1; done" ]
      }
    ]'
spec:
  hostNetwork: true
  containers:
  - name: kube-scheduler
    image: quay.io/coreos/hyperkube:v1.4.3_coreos.0
    command:
    - /hyperkube
    - scheduler
    - --master=http://127.0.0.1:8080
    - --leader-elect=true
    livenessProbe:
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10251
      initialDelaySeconds: 15
      timeoutSeconds: 1
</code></pre></div>
<p dir="auto">but it looks like init-containers only executes during POD creation. When the POD is restarted it does not execute.</p>
<p dir="auto">My bug is similar to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="172472665" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/31123" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/31123/hovercard" href="https://github.com/kubernetes/kubernetes/issues/31123">#31123</a>, but provides more data.</p>