title: Kubernetes: Pods Can't Resolve Hostnames
body: <p>I am encountering an issue with Kubernetes where my pods cannot resolve hostnames (such as <code>google.com</code> or <code>kubernetes.default</code>). </p>
  
  <p>I currently have 1 master and 1 node running on two CentOS7 instances in OpenStack. I deployed using <code>kubeadm</code>.</p>
  
  <p>Here are the versions installed:</p>
  
  <pre><code>kubeadm-1.7.3-1.x86_64
  kubectl-1.7.3-1.x86_64
  kubelet-1.7.3-1.x86_64
  kubernetes-cni-0.5.1-0.x86_64
  </code></pre>
  
  <p>The below outlines some verification steps to maybe give some insight into my problem.</p>
  
  <p>I define a busybox pod:</p>
  
  <pre><code>apiVersion: v1
  kind: Pod
  metadata:
    name: busybox
    namespace: default
  spec:
    containers:
    - image: busybox
      command:
        - sleep
        - "3600"
      imagePullPolicy: IfNotPresent
      name: busybox
    restartPolicy: Always
  </code></pre>
  
  <p>And then create the pod:</p>
  
  <pre><code>$ kubectl create -f busybox.yaml
  </code></pre>
  
  <p>Try to perform a DNS lookup of name <code>google.com</code>:</p>
  
  <pre><code>$ kubectl exec -ti busybox -- nslookup google.com
  Server:    10.96.0.10
  Address 1: 10.96.0.10
  nslookup: can't resolve 'google.com'
  </code></pre>
  
  <p>Try to perform a DNS lookup of name <code>kubernetes.default</code>:</p>
  
  <pre><code>$ kubectl exec -ti busybox -- nslookup kubernetes.default
  Server:    10.96.0.10
  Address 1: 10.96.0.10
  nslookup: can't resolve 'kubernetes.default'
  </code></pre>
  
  <p>Check if my DNS pod is running:</p>
  
  <pre><code>$ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
  NAME                        READY     STATUS    RESTARTS   AGE
  kube-dns-2425271678-k1nft   3/3       Running   9          5d
  </code></pre>
  
  <p>Check if my DNS service is up:</p>
  
  <pre><code>$ kubectl get svc --namespace=kube-system
  NAME       CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
  kube-dns   10.96.0.10   &lt;none&gt;        53/UDP,53/TCP   5d
  </code></pre>
  
  <p>Check if DNS endpoints are exposed:</p>
  
  <pre><code>$ kubectl get ep kube-dns --namespace=kube-system
  NAME       ENDPOINTS                     AGE
  kube-dns   10.244.0.5:53,10.244.0.5:53   5d
  </code></pre>
  
  <p>Check the contents of <code>/etc/resolv.conf</code> in my container:</p>
  
  <pre><code>$ kubectl exec -ti busybox -- cat /etc/resolv.conf
  nameserver 10.96.0.10
  search default.svc.cluster.local svc.cluster.local cluster.local
  options ndots:5
  </code></pre>
  
  <p>If I am understand correctly, the Kubernetes documentation states that my pods should inherit the DNS configurations of the node (or master?). However, even with just one line in it (<code>nameserver 10.92.128.40</code>), I receive the below warning when spinning up a pod:</p>
  
  <pre><code>Search Line limits were exceeded, some dns names have been omitted, the applied search line is: default.svc.cluster.local svc.cluster.local cluster.local mydomain.net anotherdomain.net yetanotherdomain.net
  </code></pre>
  
  <p>I understand there exists a known issue where only so many items can be listed in <code>/etc/resolv.conf</code>. However, where would the above search line and nameserver in my container be generated from?</p>
  
  <p>Finally here are the logs from the <code>kube-dns</code> container:</p>
  
  <pre><code>$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns
  I0817 20:54:58.445280       1 dns.go:48] version: 1.14.3-4-gee838f6
  I0817 20:54:58.452551       1 server.go:70] Using configuration read from directory: /kube-dns-config with period 10s
  I0817 20:54:58.452616       1 server.go:113] FLAG: --alsologtostderr="false"
  I0817 20:54:58.452628       1 server.go:113] FLAG: --config-dir="/kube-dns-config"
  I0817 20:54:58.452638       1 server.go:113] FLAG: --config-map=""
  I0817 20:54:58.452643       1 server.go:113] FLAG: --config-map-namespace="kube-system"
  I0817 20:54:58.452650       1 server.go:113] FLAG: --config-period="10s"
  I0817 20:54:58.452659       1 server.go:113] FLAG: --dns-bind-address="0.0.0.0"
  I0817 20:54:58.452665       1 server.go:113] FLAG: --dns-port="10053"
  I0817 20:54:58.452674       1 server.go:113] FLAG: --domain="cluster.local."
  I0817 20:54:58.452683       1 server.go:113] FLAG: --federations=""
  I0817 20:54:58.452692       1 server.go:113] FLAG: --healthz-port="8081"
  I0817 20:54:58.452698       1 server.go:113] FLAG: --initial-sync-timeout="1m0s"
  I0817 20:54:58.452704       1 server.go:113] FLAG: --kube-master-url=""
  I0817 20:54:58.452713       1 server.go:113] FLAG: --kubecfg-file=""
  I0817 20:54:58.452718       1 server.go:113] FLAG: --log-backtrace-at=":0"
  I0817 20:54:58.452727       1 server.go:113] FLAG: --log-dir=""
  I0817 20:54:58.452734       1 server.go:113] FLAG: --log-flush-frequency="5s"
  I0817 20:54:58.452741       1 server.go:113] FLAG: --logtostderr="true"
  I0817 20:54:58.452746       1 server.go:113] FLAG: --nameservers=""
  I0817 20:54:58.452752       1 server.go:113] FLAG: --stderrthreshold="2"
  I0817 20:54:58.452759       1 server.go:113] FLAG: --v="2"
  I0817 20:54:58.452765       1 server.go:113] FLAG: --version="false"
  I0817 20:54:58.452775       1 server.go:113] FLAG: --vmodule=""
  I0817 20:54:58.452856       1 server.go:176] Starting SkyDNS server (0.0.0.0:10053)
  I0817 20:54:58.453680       1 server.go:198] Skydns metrics enabled (/metrics:10055)
  I0817 20:54:58.453692       1 dns.go:147] Starting endpointsController
  I0817 20:54:58.453699       1 dns.go:150] Starting serviceController
  I0817 20:54:58.453841       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
  I0817 20:54:58.453852       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
  I0817 20:54:58.964468       1 dns.go:171] Initialized services and endpoints from apiserver
  I0817 20:54:58.964523       1 server.go:129] Setting up Healthz Handler (/readiness)
  I0817 20:54:58.964536       1 server.go:134] Setting up cache handler (/cache)
  I0817 20:54:58.964545       1 server.go:120] Status HTTP port 8081
  </code></pre>
  
  <p>The <code>dnsmasq</code> container. Disregard that it found several more nameservers than just the one I said was in my <code>resolv.conf</code>, as I did have more in their originally. I attempted to simply it by removing the extras:</p>
  
  <pre><code>$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c dnsmasq
  I0817 20:55:03.295826       1 main.go:76] opts: {{/usr/sbin/dnsmasq [-k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053] true} /etc/k8s/dns/dnsmasq-nanny 10000000000}
  I0817 20:55:03.298134       1 nanny.go:86] Starting dnsmasq [-k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053]
  I0817 20:55:03.731577       1 nanny.go:111] 
  W0817 20:55:03.731609       1 nanny.go:112] Got EOF from stdout
  I0817 20:55:03.731642       1 nanny.go:108] dnsmasq[9]: started, version 2.76 cachesize 1000
  I0817 20:55:03.731656       1 nanny.go:108] dnsmasq[9]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify
  I0817 20:55:03.731681       1 nanny.go:108] dnsmasq[9]: using nameserver 127.0.0.1#10053 for domain ip6.arpa 
  I0817 20:55:03.731689       1 nanny.go:108] dnsmasq[9]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa 
  I0817 20:55:03.731695       1 nanny.go:108] dnsmasq[9]: using nameserver 127.0.0.1#10053 for domain cluster.local 
  I0817 20:55:03.731704       1 nanny.go:108] dnsmasq[9]: reading /etc/resolv.conf
  I0817 20:55:03.731710       1 nanny.go:108] dnsmasq[9]: using nameserver 127.0.0.1#10053 for domain ip6.arpa 
  I0817 20:55:03.731717       1 nanny.go:108] dnsmasq[9]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa 
  I0817 20:55:03.731723       1 nanny.go:108] dnsmasq[9]: using nameserver 127.0.0.1#10053 for domain cluster.local 
  I0817 20:55:03.731729       1 nanny.go:108] dnsmasq[9]: using nameserver 10.92.128.40#53
  I0817 20:55:03.731735       1 nanny.go:108] dnsmasq[9]: using nameserver 10.92.128.41#53
  I0817 20:55:03.731741       1 nanny.go:108] dnsmasq[9]: using nameserver 10.95.207.66#53
  I0817 20:55:03.731747       1 nanny.go:108] dnsmasq[9]: read /etc/hosts - 7 addresses
  </code></pre>
  
  <p>And the <code>sidecar</code> container:</p>
  
  <pre><code>$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c sidecar
  ERROR: logging before flag.Parse: I0817 20:55:04.488391       1 main.go:48] Version v1.14.3-4-gee838f6
  ERROR: logging before flag.Parse: I0817 20:55:04.488612       1 server.go:45] Starting server (options {DnsMasqPort:53 DnsMasqAddr:127.0.0.1 DnsMasqPollIntervalMs:5000 Probes:[{Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1} {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}] PrometheusAddr:0.0.0.0 PrometheusPort:10054 PrometheusPath:/metrics PrometheusNamespace:kubedns})
  ERROR: logging before flag.Parse: I0817 20:55:04.488667       1 dnsprobe.go:75] Starting dnsProbe {Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}
  ERROR: logging before flag.Parse: I0817 20:55:04.488766       1 dnsprobe.go:75] Starting dnsProbe {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}
  </code></pre>
  
  <p>I have mostly been reading the documentation provided <a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/" rel="noreferrer">here</a>. Any direction, insight or things to try would be much appreciated. </p>