<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): vSphere, "message": "no endpoints available for service "kube-dns"", kube-dns, etc -- similar issues include: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="149190989" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/24407" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/24407/hovercard" href="https://github.com/kubernetes/kubernetes/issues/24407">#24407</a> however it's not quite the same</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): Bug</p>
<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.0&quot;, GitCommit:&quot;283137936a498aed572ee22af6774b6fb6e9fd94&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-07-01T19:26:38Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;darwin/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;4&quot;, GitVersion:&quot;v1.4.0&quot;, GitCommit:&quot;a16c0a7f71a6f93c7e0f222d961f4675cd97a46b&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-09-26T18:10:32Z&quot;, GoVersion:&quot;go1.6.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"283137936a498aed572ee22af6774b6fb6e9fd94", GitTreeState:"clean", BuildDate:"2016-07-01T19:26:38Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: vSphere</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): Using provided vmdk</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): <code class="notranslate">Linux kubernetes-minion-4 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u2 (2016-01-02) x86_64 GNU/Linux</code> and <code class="notranslate">Linux kubernetes-master 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u2 (2016-01-02) x86_64 GNU/Linux</code></li>
<li><strong>Install tools</strong>: GOVC + the <code class="notranslate">cluster/vsphere/config-default.sh</code> script</li>
</ul>
<p dir="auto"><strong>What happened</strong>: Our environment utilizes the RFC1918 10.0.0.0/8 block, so I edited <code class="notranslate">cluster/vsphere/config-default.sh</code> to utilize an address space we are not using. This is my current configuration:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="NODE_IP_RANGES=&quot;192.168.0.0/17&quot;
MASTER_IP_RANGE=&quot;${MASTER_IP_RANGE:-192.168.128.0/17}&quot;
SERVICE_CLUSTER_IP_RANGE=&quot;192.168.120.0/21&quot;  # formerly PORTAL_NET
DNS_SERVER_IP=&quot;192.168.120.120&quot;"><pre class="notranslate"><code class="notranslate">NODE_IP_RANGES="192.168.0.0/17"
MASTER_IP_RANGE="${MASTER_IP_RANGE:-192.168.128.0/17}"
SERVICE_CLUSTER_IP_RANGE="192.168.120.0/21"  # formerly PORTAL_NET
DNS_SERVER_IP="192.168.120.120"
</code></pre></div>
<p dir="auto">After the deployment is finished, I am able to login to the dashboard via the URL provided via <code class="notranslate">kubectl cluster-info</code>, however, when I check the DNS server I see:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt; curl -sk https://admin:password@10.248.30.36/api/v1/proxy/namespaces/kube-system/services/kube-dns
{
  &quot;kind&quot;: &quot;Status&quot;,
  &quot;apiVersion&quot;: &quot;v1&quot;,
  &quot;metadata&quot;: {},
  &quot;status&quot;: &quot;Failure&quot;,
  &quot;message&quot;: &quot;no endpoints available for service \&quot;kube-dns\&quot;&quot;,
  &quot;reason&quot;: &quot;ServiceUnavailable&quot;,
  &quot;code&quot;: 503
}"><pre class="notranslate"><code class="notranslate">&gt; curl -sk https://admin:password@10.248.30.36/api/v1/proxy/namespaces/kube-system/services/kube-dns
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "no endpoints available for service \"kube-dns\"",
  "reason": "ServiceUnavailable",
  "code": 503
}
</code></pre></div>
<p dir="auto">When I list services, I see:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt; kubectl get svc --namespace=kube-system
NAME                   CLUSTER-IP        EXTERNAL-IP   PORT(S)         AGE
kube-dns               192.168.120.120   &lt;none&gt;        53/UDP,53/TCP   50m
kubernetes-dashboard   192.168.127.194   &lt;none&gt;        80/TCP          50m"><pre class="notranslate"><code class="notranslate">&gt; kubectl get svc --namespace=kube-system
NAME                   CLUSTER-IP        EXTERNAL-IP   PORT(S)         AGE
kube-dns               192.168.120.120   &lt;none&gt;        53/UDP,53/TCP   50m
kubernetes-dashboard   192.168.127.194   &lt;none&gt;        80/TCP          50m
</code></pre></div>
<p dir="auto">When I start a container and try to ping kube-dns, I am unable to:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt; kubectl exec -ti testing-network-2429464084-8jauq /bin/bash
root@testing-network-2429464084-8jauq:/# ping 192.168.120.120
PING 192.168.120.120 (192.168.120.120) 56(84) bytes of data.
^C
--- 192.168.120.120 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4032ms"><pre class="notranslate"><code class="notranslate">&gt; kubectl exec -ti testing-network-2429464084-8jauq /bin/bash
root@testing-network-2429464084-8jauq:/# ping 192.168.120.120
PING 192.168.120.120 (192.168.120.120) 56(84) bytes of data.
^C
--- 192.168.120.120 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4032ms
</code></pre></div>
<p dir="auto">I can ping external IP addresses:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# ping 216.58.192.142
PING 216.58.192.142 (216.58.192.142) 56(84) bytes of data.
64 bytes from 216.58.192.142: icmp_seq=1 ttl=50 time=52.0 ms
64 bytes from 216.58.192.142: icmp_seq=2 ttl=50 time=51.9 ms
^C
--- 216.58.192.142 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 51.968/51.999/52.031/0.230 ms"><pre class="notranslate"><code class="notranslate"># ping 216.58.192.142
PING 216.58.192.142 (216.58.192.142) 56(84) bytes of data.
64 bytes from 216.58.192.142: icmp_seq=1 ttl=50 time=52.0 ms
64 bytes from 216.58.192.142: icmp_seq=2 ttl=50 time=51.9 ms
^C
--- 216.58.192.142 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 51.968/51.999/52.031/0.230 ms
</code></pre></div>
<p dir="auto">Here's the configuration on each minion and the master:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Master - IP: 10.248.30.36
kube@kubernetes-master:~$ ip ro sh
default via 10.248.30.1 dev eth0
10.248.30.0/24 dev eth0  proto kernel  scope link  src 10.248.30.36
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1
172.18.0.0/16 via 10.248.30.39 dev eth0
192.168.0.0/24 via 10.248.30.38 dev eth0
192.168.1.0/24 via 10.248.30.40 dev eth0
192.168.2.0/24 via 10.248.30.37 dev eth0
192.168.128.0/17 dev cbr0  proto kernel  scope link  src 192.168.128.1


Minion 1 - IP: 10.248.30.38
kube@kubernetes-minion-1:~$ ip ro sh
default via 10.248.30.1 dev eth0
10.248.30.0/24 dev eth0  proto kernel  scope link  src 10.248.30.38
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1
172.18.0.0/16 via 10.248.30.39 dev eth0
192.168.0.0/24 dev cbr0  proto kernel  scope link  src 192.168.0.1
192.168.1.0/24 via 10.248.30.40 dev eth0
192.168.2.0/24 via 10.248.30.37 dev eth0


Minion 2 - IP: 10.248.30.40
kube@kubernetes-minion-2:~$ ip ro sh
default via 10.248.30.1 dev eth0
10.248.30.0/24 dev eth0  proto kernel  scope link  src 10.248.30.40
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1
172.18.0.0/16 via 10.248.30.39 dev eth0
192.168.0.0/24 via 10.248.30.38 dev eth0
192.168.1.0/24 dev cbr0  proto kernel  scope link  src 192.168.1.1
192.168.2.0/24 via 10.248.30.37 dev eth0

Minion 2 - IP: 10.248.30.37
kube@kubernetes-minion-3:~$ ip ro sh
default via 10.248.30.1 dev eth0
10.248.30.0/24 dev eth0  proto kernel  scope link  src 10.248.30.37
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1
172.18.0.0/16 via 10.248.30.39 dev eth0
192.168.0.0/24 via 10.248.30.38 dev eth0
192.168.1.0/24 via 10.248.30.40 dev eth0
192.168.2.0/24 dev cbr0  proto kernel  scope link  src 192.168.2.1


Minion 4 - IP: 10.248.30.39
kube@kubernetes-minion-4:~$ ip ro sh
default via 10.248.30.1 dev eth0
10.248.30.0/24 dev eth0  proto kernel  scope link  src 10.248.30.39
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1
192.168.0.0/24 via 10.248.30.38 dev eth0
192.168.1.0/24 via 10.248.30.40 dev eth0
192.168.2.0/24 via 10.248.30.37 dev eth0
192.168.3.0/24 dev cbr0  proto kernel  scope link  src 192.168.3.1"><pre class="notranslate"><code class="notranslate">Master - IP: 10.248.30.36
kube@kubernetes-master:~$ ip ro sh
default via 10.248.30.1 dev eth0
10.248.30.0/24 dev eth0  proto kernel  scope link  src 10.248.30.36
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1
172.18.0.0/16 via 10.248.30.39 dev eth0
192.168.0.0/24 via 10.248.30.38 dev eth0
192.168.1.0/24 via 10.248.30.40 dev eth0
192.168.2.0/24 via 10.248.30.37 dev eth0
192.168.128.0/17 dev cbr0  proto kernel  scope link  src 192.168.128.1


Minion 1 - IP: 10.248.30.38
kube@kubernetes-minion-1:~$ ip ro sh
default via 10.248.30.1 dev eth0
10.248.30.0/24 dev eth0  proto kernel  scope link  src 10.248.30.38
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1
172.18.0.0/16 via 10.248.30.39 dev eth0
192.168.0.0/24 dev cbr0  proto kernel  scope link  src 192.168.0.1
192.168.1.0/24 via 10.248.30.40 dev eth0
192.168.2.0/24 via 10.248.30.37 dev eth0


Minion 2 - IP: 10.248.30.40
kube@kubernetes-minion-2:~$ ip ro sh
default via 10.248.30.1 dev eth0
10.248.30.0/24 dev eth0  proto kernel  scope link  src 10.248.30.40
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1
172.18.0.0/16 via 10.248.30.39 dev eth0
192.168.0.0/24 via 10.248.30.38 dev eth0
192.168.1.0/24 dev cbr0  proto kernel  scope link  src 192.168.1.1
192.168.2.0/24 via 10.248.30.37 dev eth0

Minion 2 - IP: 10.248.30.37
kube@kubernetes-minion-3:~$ ip ro sh
default via 10.248.30.1 dev eth0
10.248.30.0/24 dev eth0  proto kernel  scope link  src 10.248.30.37
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1
172.18.0.0/16 via 10.248.30.39 dev eth0
192.168.0.0/24 via 10.248.30.38 dev eth0
192.168.1.0/24 via 10.248.30.40 dev eth0
192.168.2.0/24 dev cbr0  proto kernel  scope link  src 192.168.2.1


Minion 4 - IP: 10.248.30.39
kube@kubernetes-minion-4:~$ ip ro sh
default via 10.248.30.1 dev eth0
10.248.30.0/24 dev eth0  proto kernel  scope link  src 10.248.30.39
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1
192.168.0.0/24 via 10.248.30.38 dev eth0
192.168.1.0/24 via 10.248.30.40 dev eth0
192.168.2.0/24 via 10.248.30.37 dev eth0
192.168.3.0/24 dev cbr0  proto kernel  scope link  src 192.168.3.1
</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>: I expect containers can reach ping eachother.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible): I've reproduced this problem using both 172.16.0.0/16 addresses as well as the 192.168.0.0/16 addresses.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>: I followed the documentation here: <a href="http://kubernetes.io/docs/getting-started-guides/vsphere/" rel="nofollow">http://kubernetes.io/docs/getting-started-guides/vsphere/</a> and the only change I made was to the network configuration.</p>
<p dir="auto">The <code class="notranslate">test-cmd</code> script currently does not test resizing replication controllers; this is a central use-case and should have coverage in that script.</p>