<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): "lock is held by"  "has not yet expired"</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):<br>
tested on 1.3.5</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.0&quot;, GitCommit:&quot;283137936a498aed572ee22af6774b6fb6e9fd94&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-07-01T19:26:38Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.5+coreos.1&quot;, GitCommit:&quot;4fc64a21f8a593d97b75296acd1e979391518b28&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-08-22T17:15:28Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"283137936a498aed572ee22af6774b6fb6e9fd94", GitTreeState:"clean", BuildDate:"2016-07-01T19:26:38Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.5+coreos.1", GitCommit:"4fc64a21f8a593d97b75296acd1e979391518b28", GitTreeState:"clean", BuildDate:"2016-08-22T17:15:28Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto">and 1.3.8 (upgraded today)</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.0&quot;, GitCommit:&quot;283137936a498aed572ee22af6774b6fb6e9fd94&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-07-01T19:26:38Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.8+coreos.1&quot;, GitCommit:&quot;6b29caf6507ce9989984720212fd3e3341069e06&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-09-30T16:54:50Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"283137936a498aed572ee22af6774b6fb6e9fd94", GitTreeState:"clean", BuildDate:"2016-07-01T19:26:38Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.8+coreos.1", GitCommit:"6b29caf6507ce9989984720212fd3e3341069e06", GitTreeState:"clean", BuildDate:"2016-09-30T16:54:50Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li>Hardware: vmware guest</li>
<li>OS: CoreOs 1122.2.0</li>
<li>kernel: 4.7.0-coreos</li>
<li>masters: 3</li>
</ul>
<p dir="auto">/etc/kubernetes/manifests/kube-scheduler.yaml looks like this:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apiVersion: v1
kind: Pod
metadata:
  name: kube-scheduler
  namespace: kube-system
  labels:
    app: kube-scheduler
spec:
  hostNetwork: true
  containers:
  - name: kube-scheduler
    image: quay.io/coreos/hyperkube:v1.3.5_coreos.1
    command:
    - /hyperkube
    - scheduler
    - --master=http://127.0.0.1:8080
    - --leader-elect=true
    - --v=2
    livenessProbe:
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10251
      initialDelaySeconds: 15
      timeoutSeconds: 1"><pre class="notranslate"><code class="notranslate">apiVersion: v1
kind: Pod
metadata:
  name: kube-scheduler
  namespace: kube-system
  labels:
    app: kube-scheduler
spec:
  hostNetwork: true
  containers:
  - name: kube-scheduler
    image: quay.io/coreos/hyperkube:v1.3.5_coreos.1
    command:
    - /hyperkube
    - scheduler
    - --master=http://127.0.0.1:8080
    - --leader-elect=true
    - --v=2
    livenessProbe:
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10251
      initialDelaySeconds: 15
      timeoutSeconds: 1
</code></pre></div>
<p dir="auto">/etc/kubernetes/manifests/kube-controller-manager.yaml looks like this:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apiVersion: v1
kind: Pod
metadata:
  name: kube-controller-manager
  namespace: kube-system
  labels:
    app: kube-controller-manager
spec:
  hostNetwork: true
  containers:
  - name: kube-controller-manager
    image: quay.io/coreos/hyperkube:v1.3.5_coreos.1
    command:
    - /hyperkube
    - controller-manager
    - --master=http://127.0.0.1:8080
    - --leader-elect=true 
    - --service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
    - --root-ca-file=/etc/kubernetes/ssl/ca.pem
    - --v=2
    livenessProbe:
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10252
      initialDelaySeconds: 15
      timeoutSeconds: 1
    volumeMounts:
    - mountPath: /etc/kubernetes/ssl
      name: ssl-certs-kubernetes
      readOnly: true
    - mountPath: /etc/ssl/certs
      name: ssl-certs-host
      readOnly: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/ssl
    name: ssl-certs-kubernetes
  - hostPath:
      path: /usr/share/ca-certificates
    name: ssl-certs-host
"><pre class="notranslate"><code class="notranslate">apiVersion: v1
kind: Pod
metadata:
  name: kube-controller-manager
  namespace: kube-system
  labels:
    app: kube-controller-manager
spec:
  hostNetwork: true
  containers:
  - name: kube-controller-manager
    image: quay.io/coreos/hyperkube:v1.3.5_coreos.1
    command:
    - /hyperkube
    - controller-manager
    - --master=http://127.0.0.1:8080
    - --leader-elect=true 
    - --service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
    - --root-ca-file=/etc/kubernetes/ssl/ca.pem
    - --v=2
    livenessProbe:
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10252
      initialDelaySeconds: 15
      timeoutSeconds: 1
    volumeMounts:
    - mountPath: /etc/kubernetes/ssl
      name: ssl-certs-kubernetes
      readOnly: true
    - mountPath: /etc/ssl/certs
      name: ssl-certs-host
      readOnly: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/ssl
    name: ssl-certs-kubernetes
  - hostPath:
      path: /usr/share/ca-certificates
    name: ssl-certs-host

</code></pre></div>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">The logs from kube-scheduler and kube-controller-manager are spamming with:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="I1012 07:13:30.700862       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:33.263266       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:36.944429       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:40.486944       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:44.772153       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:47.186971       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:49.914362       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:52.353564       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:56.181010       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:59.263994       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:03.553559       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:06.325838       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:10.595729       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:13.347343       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:17.330717       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:21.505816       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:23.950507       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:26.892771       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:30.914393       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:34.408769       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:38.544271       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:40.645166       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:43.919125       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:47.663210       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:51.768576       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:55.748494       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:00.116892       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:03.489400       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:06.465890       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:09.703411       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:13.463524       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:15.607621       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:17.896426       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:22.087427       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:26.217274       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:30.269665       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:33.637867       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:36.712575       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:40.783867       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:44.372188       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:47.605168       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:50.434502       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:53.216699       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:57.015595       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:59.152636       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:16:01.301530       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired"><pre class="notranslate"><code class="notranslate">I1012 07:13:30.700862       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:33.263266       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:36.944429       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:40.486944       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:44.772153       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:47.186971       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:49.914362       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:52.353564       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:56.181010       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:59.263994       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:03.553559       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:06.325838       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:10.595729       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:13.347343       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:17.330717       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:21.505816       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:23.950507       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:26.892771       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:30.914393       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:34.408769       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:38.544271       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:40.645166       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:43.919125       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:47.663210       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:51.768576       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:55.748494       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:00.116892       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:03.489400       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:06.465890       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:09.703411       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:13.463524       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:15.607621       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:17.896426       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:22.087427       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:26.217274       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:30.269665       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:33.637867       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:36.712575       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:40.783867       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:44.372188       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:47.605168       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:50.434502       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:53.216699       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:57.015595       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:59.152636       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:16:01.301530       1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
</code></pre></div>
<p dir="auto">The node holding the lock only logs this:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="I1012 07:27:20.982437       1 leaderelection.go:215] sucessfully acquired lease kube-system/kube-scheduler"><pre class="notranslate"><code class="notranslate">I1012 07:27:20.982437       1 leaderelection.go:215] sucessfully acquired lease kube-system/kube-scheduler
</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
If this is expected behaviour i would like to be able to supress those messages with --v=2</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):<br>
I think it would be sufficient to run scheduler or controller-manager with <code class="notranslate">--leader-elect?true</code>.</p>
<p dir="auto">etcdctl get registry/services/endpoints/kube-system/kube-controller-manager looks like this:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
    &quot;kind&quot;: &quot;Endpoints&quot;,
    &quot;apiVersion&quot;: &quot;v1&quot;,
    &quot;metadata&quot;: {
        &quot;name&quot;: &quot;kube-controller-manager&quot;,
        &quot;namespace&quot;: &quot;kube-system&quot;,
        &quot;selfLink&quot;: &quot;/api/v1/namespaces/kube-system/endpoints/kube-controller-manager&quot;,
        &quot;uid&quot;: &quot;b9ee7a6f-8f82-11e6-b186-0050568802e1&quot;,
        &quot;creationTimestamp&quot;: &quot;2016-10-11T07:17:10Z&quot;,
        &quot;annotations&quot;: {
            &quot;control-plane.alpha.kubernetes.io/leader&quot;: &quot;{\&quot;holderIdentity\&quot;:\&quot;utv-kube03\&quot;,\&quot;leaseDurationSeconds\&quot;:15,\&quot;acquireTime\&quot;:\&quot;2016-10-11T07:17:10Z\&quot;,\&quot;renewTime\&quot;:\&quot;2016-10-12T07:18:54Z\&quot;,\&quot;leaderTransitions\&quot;:0}&quot;
        }
    },
    &quot;subsets&quot;: []
}"><pre class="notranslate"><code class="notranslate">{
    "kind": "Endpoints",
    "apiVersion": "v1",
    "metadata": {
        "name": "kube-controller-manager",
        "namespace": "kube-system",
        "selfLink": "/api/v1/namespaces/kube-system/endpoints/kube-controller-manager",
        "uid": "b9ee7a6f-8f82-11e6-b186-0050568802e1",
        "creationTimestamp": "2016-10-11T07:17:10Z",
        "annotations": {
            "control-plane.alpha.kubernetes.io/leader": "{\"holderIdentity\":\"utv-kube03\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2016-10-11T07:17:10Z\",\"renewTime\":\"2016-10-12T07:18:54Z\",\"leaderTransitions\":0}"
        }
    },
    "subsets": []
}
</code></pre></div>
<p dir="auto">The cluster seems healthy and are scheduling pods correctly.</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<h2 dir="auto">kube-dns, kubeadm, headless service, service</h2>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="kubectl version
Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;4&quot;, GitVersion:&quot;v1.4.0&quot;, GitCommit:&quot;a16c0a7f71a6f93c7e0f222d961f4675cd97a46b&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-09-26T18:16:57Z&quot;, GoVersion:&quot;go1.6.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;darwin/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;4&quot;, GitVersion:&quot;v1.4.0&quot;, GitCommit:&quot;a16c0a7f71a6f93c7e0f222d961f4675cd97a46b&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-09-26T18:10:32Z&quot;, GoVersion:&quot;go1.6.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate">kubectl version
Client Version: version.Info{Major:<span class="pl-s"><span class="pl-pds">"</span>1<span class="pl-pds">"</span></span>, Minor:<span class="pl-s"><span class="pl-pds">"</span>4<span class="pl-pds">"</span></span>, GitVersion:<span class="pl-s"><span class="pl-pds">"</span>v1.4.0<span class="pl-pds">"</span></span>, GitCommit:<span class="pl-s"><span class="pl-pds">"</span>a16c0a7f71a6f93c7e0f222d961f4675cd97a46b<span class="pl-pds">"</span></span>, GitTreeState:<span class="pl-s"><span class="pl-pds">"</span>clean<span class="pl-pds">"</span></span>, BuildDate:<span class="pl-s"><span class="pl-pds">"</span>2016-09-26T18:16:57Z<span class="pl-pds">"</span></span>, GoVersion:<span class="pl-s"><span class="pl-pds">"</span>go1.6.3<span class="pl-pds">"</span></span>, Compiler:<span class="pl-s"><span class="pl-pds">"</span>gc<span class="pl-pds">"</span></span>, Platform:<span class="pl-s"><span class="pl-pds">"</span>darwin/amd64<span class="pl-pds">"</span></span>}
Server Version: version.Info{Major:<span class="pl-s"><span class="pl-pds">"</span>1<span class="pl-pds">"</span></span>, Minor:<span class="pl-s"><span class="pl-pds">"</span>4<span class="pl-pds">"</span></span>, GitVersion:<span class="pl-s"><span class="pl-pds">"</span>v1.4.0<span class="pl-pds">"</span></span>, GitCommit:<span class="pl-s"><span class="pl-pds">"</span>a16c0a7f71a6f93c7e0f222d961f4675cd97a46b<span class="pl-pds">"</span></span>, GitTreeState:<span class="pl-s"><span class="pl-pds">"</span>clean<span class="pl-pds">"</span></span>, BuildDate:<span class="pl-s"><span class="pl-pds">"</span>2016-09-26T18:10:32Z<span class="pl-pds">"</span></span>, GoVersion:<span class="pl-s"><span class="pl-pds">"</span>go1.6.3<span class="pl-pds">"</span></span>, Compiler:<span class="pl-s"><span class="pl-pds">"</span>gc<span class="pl-pds">"</span></span>, Platform:<span class="pl-s"><span class="pl-pds">"</span>linux/amd64<span class="pl-pds">"</span></span>}</pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: hardware x64</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):</li>
</ul>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=" cat /etc/os-release
NAME=&quot;CentOS Linux&quot;
VERSION=&quot;7 (Core)&quot;
ID=&quot;centos&quot;
ID_LIKE=&quot;rhel fedora&quot;
VERSION_ID=&quot;7&quot;
PRETTY_NAME=&quot;CentOS Linux 7 (Core)&quot;
ANSI_COLOR=&quot;0;31&quot;
CPE_NAME=&quot;cpe:/o:centos:centos:7&quot;
HOME_URL=&quot;https://www.centos.org/&quot;
BUG_REPORT_URL=&quot;https://bugs.centos.org/&quot;

CENTOS_MANTISBT_PROJECT=&quot;CentOS-7&quot;
CENTOS_MANTISBT_PROJECT_VERSION=&quot;7&quot;
REDHAT_SUPPORT_PRODUCT=&quot;centos&quot;
REDHAT_SUPPORT_PRODUCT_VERSION=&quot;7&quot;"><pre class="notranslate"> cat /etc/os-release
NAME=<span class="pl-s"><span class="pl-pds">"</span>CentOS Linux<span class="pl-pds">"</span></span>
VERSION=<span class="pl-s"><span class="pl-pds">"</span>7 (Core)<span class="pl-pds">"</span></span>
ID=<span class="pl-s"><span class="pl-pds">"</span>centos<span class="pl-pds">"</span></span>
ID_LIKE=<span class="pl-s"><span class="pl-pds">"</span>rhel fedora<span class="pl-pds">"</span></span>
VERSION_ID=<span class="pl-s"><span class="pl-pds">"</span>7<span class="pl-pds">"</span></span>
PRETTY_NAME=<span class="pl-s"><span class="pl-pds">"</span>CentOS Linux 7 (Core)<span class="pl-pds">"</span></span>
ANSI_COLOR=<span class="pl-s"><span class="pl-pds">"</span>0;31<span class="pl-pds">"</span></span>
CPE_NAME=<span class="pl-s"><span class="pl-pds">"</span>cpe:/o:centos:centos:7<span class="pl-pds">"</span></span>
HOME_URL=<span class="pl-s"><span class="pl-pds">"</span>https://www.centos.org/<span class="pl-pds">"</span></span>
BUG_REPORT_URL=<span class="pl-s"><span class="pl-pds">"</span>https://bugs.centos.org/<span class="pl-pds">"</span></span>

CENTOS_MANTISBT_PROJECT=<span class="pl-s"><span class="pl-pds">"</span>CentOS-7<span class="pl-pds">"</span></span>
CENTOS_MANTISBT_PROJECT_VERSION=<span class="pl-s"><span class="pl-pds">"</span>7<span class="pl-pds">"</span></span>
REDHAT_SUPPORT_PRODUCT=<span class="pl-s"><span class="pl-pds">"</span>centos<span class="pl-pds">"</span></span>
REDHAT_SUPPORT_PRODUCT_VERSION=<span class="pl-s"><span class="pl-pds">"</span>7<span class="pl-pds">"</span></span></pre></div>
<ul dir="auto">
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):</li>
</ul>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="uname -a
Linux panel4.local.minirz.de 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux"><pre class="notranslate">uname -a
Linux panel4.local.minirz.de 3.10.0-327.18.2.el7.x86_64 <span class="pl-c"><span class="pl-c">#</span>1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux</span></pre></div>
<ul dir="auto">
<li><strong>Install tools</strong>:</li>
</ul>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="kubeadm and weave"><pre class="notranslate">kubeadm and weave</pre></div>
<ul dir="auto">
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
A headless service doesn't resolve to an A RR</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl get endpoints 
NAME         ENDPOINTS                                      AGE
api                                                         2h
registry     10.44.0.2:7866,10.44.0.2:7866,10.44.0.2:7876   2h
webconsole                                                  2h

# Use default DNS Server fails
$ kubectl exec -ti debug -- nslookup registry
Server:    100.64.0.10
Address 1: 100.64.0.10

nslookup: can't resolve 'registry'

 kubectl get endpoints --namespace=kube-system
NAME                      ENDPOINTS                   AGE
kube-controller-manager   &lt;none&gt;                      2h
kube-dns                  10.36.0.3:53,10.36.0.3:53   4h
kube-scheduler            &lt;none&gt;                      2h
kubernetes-dashboard      10.44.0.3:9090              4h

# Use the kube-dns Pod IP works
$ kubectl exec -ti debug -- nslookup registry 10.36.0.3
Server:    10.36.0.3
Address 1: 10.36.0.3 kube-dns-2247936740-s5lu1

Name:      registry
Address 1: 10.44.0.4 debug"><pre class="notranslate"><code class="notranslate">$ kubectl get endpoints 
NAME         ENDPOINTS                                      AGE
api                                                         2h
registry     10.44.0.2:7866,10.44.0.2:7866,10.44.0.2:7876   2h
webconsole                                                  2h

# Use default DNS Server fails
$ kubectl exec -ti debug -- nslookup registry
Server:    100.64.0.10
Address 1: 100.64.0.10

nslookup: can't resolve 'registry'

 kubectl get endpoints --namespace=kube-system
NAME                      ENDPOINTS                   AGE
kube-controller-manager   &lt;none&gt;                      2h
kube-dns                  10.36.0.3:53,10.36.0.3:53   4h
kube-scheduler            &lt;none&gt;                      2h
kubernetes-dashboard      10.44.0.3:9090              4h

# Use the kube-dns Pod IP works
$ kubectl exec -ti debug -- nslookup registry 10.36.0.3
Server:    10.36.0.3
Address 1: 10.36.0.3 kube-dns-2247936740-s5lu1

Name:      registry
Address 1: 10.44.0.4 debug
</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
Getting the A RR for all endpoints of this headless service.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">Create a cluster with kubeadm + weave then create a headless service:</p>
<div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="apiVersion: v1
kind: Service
metadata:
  name: registry
spec:
  clusterIP: &quot;None&quot; # headless service =&gt; kube-dns will return pod IPs
  ports:
    - name rpc
      port: 7866
      protocol: TCP
  selector:
    role: registry"><pre class="notranslate"><span class="pl-ent">apiVersion</span>: <span class="pl-c1">v1</span>
<span class="pl-ent">kind</span>: <span class="pl-s">Service</span>
<span class="pl-ent">metadata</span>:
  <span class="pl-ent">name</span>: <span class="pl-s">registry</span>
<span class="pl-ent">spec</span>:
  <span class="pl-ent">clusterIP</span>: <span class="pl-s"><span class="pl-pds">"</span>None<span class="pl-pds">"</span></span> <span class="pl-c"><span class="pl-c">#</span> headless service =&gt; kube-dns will return pod IPs</span>
  <span class="pl-ent">ports</span>:
    - <span class="pl-s">name rpc</span>
      <span class="pl-ent">port</span>: <span class="pl-c1">7866</span>
      <span class="pl-ent">protocol</span>: <span class="pl-s">TCP</span>
  <span class="pl-ent">selector</span>:
    <span class="pl-ent">role</span>: <span class="pl-s">registry</span></pre></div>
<p dir="auto">and try to resolve it.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">On the node running the debug pod:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="ifconfig weave
weave: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1410
        inet 10.32.0.1  netmask 255.240.0.0  broadcast 0.0.0.0
        inet6 fe80::c88:2dff:fe1a:e66c  prefixlen 64  scopeid 0x20&lt;link&gt;
        ether 0e:88:2d:1a:e6:6c  txqueuelen 0  (Ethernet)
        RX packets 366  bytes 21068 (20.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 303  bytes 23660 (23.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ifconfig | grep 100.64"><pre class="notranslate">ifconfig weave
weave: flags=<span class="pl-k">4163&lt;</span>UP,BROADCAST,RUNNING,MULTICAST<span class="pl-k">&gt;</span>  mtu 1410
        inet 10.32.0.1  netmask 255.240.0.0  broadcast 0.0.0.0
        inet6 fe80::c88:2dff:fe1a:e66c  prefixlen 64  scopeid 0x<span class="pl-k">20&lt;</span>link<span class="pl-k">&gt;</span>
        ether 0e:88:2d:1a:e6:6c  txqueuelen 0  (Ethernet)
        RX packets 366  bytes 21068 (20.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 303  bytes 23660 (23.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ifconfig <span class="pl-k">|</span> grep 100.64</pre></div>
<p dir="auto">and the iptables output:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="[root@panel4 ~]# iptables-save | grep 100.64
-A KUBE-SERVICES -d 100.64.0.1/32 -p tcp -m comment --comment &quot;default/kubernetes:https cluster IP&quot; -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -d 100.64.0.10/32 -p udp -m comment --comment &quot;kube-system/kube-dns:dns cluster IP&quot; -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 100.64.0.10/32 -p tcp -m comment --comment &quot;kube-system/kube-dns:dns-tcp cluster IP&quot; -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
[root@panel4 ~]# iptables-save | grep KUBE-SVC-TCOU7JCQXEZGVUNU
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
-A KUBE-SERVICES -d 100.64.0.10/32 -p udp -m comment --comment &quot;kube-system/kube-dns:dns cluster IP&quot; -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment &quot;kube-system/kube-dns:dns&quot; -j KUBE-SEP-6RTHVOJVKE2IJ2GA
[root@panel4 ~]# iptables-save | grep KUBE-SEP-6RTHVOJVKE2IJ2GA
:KUBE-SEP-6RTHVOJVKE2IJ2GA - [0:0]
-A KUBE-SEP-6RTHVOJVKE2IJ2GA -s 10.36.0.3/32 -m comment --comment &quot;kube-system/kube-dns:dns&quot; -j KUBE-MARK-MASQ
-A KUBE-SEP-6RTHVOJVKE2IJ2GA -p udp -m comment --comment &quot;kube-system/kube-dns:dns&quot; -m udp -j DNAT --to-destination 10.36.0.3:53
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment &quot;kube-system/kube-dns:dns&quot; -j KUBE-SEP-6RTHVOJVKE2IJ2GA"><pre class="notranslate">[root@panel4 <span class="pl-k">~</span>]<span class="pl-c"><span class="pl-c">#</span> iptables-save | grep 100.64</span>
-A KUBE-SERVICES -d 100.64.0.1/32 -p tcp -m comment --comment <span class="pl-s"><span class="pl-pds">"</span>default/kubernetes:https cluster IP<span class="pl-pds">"</span></span> -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -d 100.64.0.10/32 -p udp -m comment --comment <span class="pl-s"><span class="pl-pds">"</span>kube-system/kube-dns:dns cluster IP<span class="pl-pds">"</span></span> -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 100.64.0.10/32 -p tcp -m comment --comment <span class="pl-s"><span class="pl-pds">"</span>kube-system/kube-dns:dns-tcp cluster IP<span class="pl-pds">"</span></span> -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
[root@panel4 <span class="pl-k">~</span>]<span class="pl-c"><span class="pl-c">#</span> iptables-save | grep KUBE-SVC-TCOU7JCQXEZGVUNU</span>
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
-A KUBE-SERVICES -d 100.64.0.10/32 -p udp -m comment --comment <span class="pl-s"><span class="pl-pds">"</span>kube-system/kube-dns:dns cluster IP<span class="pl-pds">"</span></span> -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment <span class="pl-s"><span class="pl-pds">"</span>kube-system/kube-dns:dns<span class="pl-pds">"</span></span> -j KUBE-SEP-6RTHVOJVKE2IJ2GA
[root@panel4 <span class="pl-k">~</span>]<span class="pl-c"><span class="pl-c">#</span> iptables-save | grep KUBE-SEP-6RTHVOJVKE2IJ2GA</span>
:KUBE-SEP-6RTHVOJVKE2IJ2GA - [0:0]
-A KUBE-SEP-6RTHVOJVKE2IJ2GA -s 10.36.0.3/32 -m comment --comment <span class="pl-s"><span class="pl-pds">"</span>kube-system/kube-dns:dns<span class="pl-pds">"</span></span> -j KUBE-MARK-MASQ
-A KUBE-SEP-6RTHVOJVKE2IJ2GA -p udp -m comment --comment <span class="pl-s"><span class="pl-pds">"</span>kube-system/kube-dns:dns<span class="pl-pds">"</span></span> -m udp -j DNAT --to-destination 10.36.0.3:53
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment <span class="pl-s"><span class="pl-pds">"</span>kube-system/kube-dns:dns<span class="pl-pds">"</span></span> -j KUBE-SEP-6RTHVOJVKE2IJ2GA</pre></div>
<p dir="auto">Maybe related: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="179839447" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/33671" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/33671/hovercard?comment_id=250576294&amp;comment_type=issue_comment" href="https://github.com/kubernetes/kubernetes/issues/33671#issuecomment-250576294">#33671 (comment)</a></p>
<p dir="auto">FYI: <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/bboreham/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/bboreham">@bboreham</a></p>