title: Kubernetes DNS lookg not working from worker node - connection timed out; no servers could be reached
body: <p>I have build new Kubernetes cluster <code>v1.20.1</code> single master and single node with Calico CNI.</p>
  <p>I deployed the <code>busybox </code>pod in default namespace.</p>
  <pre><code># kubectl get pods busybox -o wide
  NAME      READY   STATUS    RESTARTS   AGE   IP             NODE        NOMINATED NODE   READINESS GATES
  busybox   1/1     Running   0          12m   10.203.0.129   node02   &lt;none&gt;           &lt;none&gt;
  
   
  </code></pre>
  <p><strong>nslookup not working</strong></p>
  <pre><code>kubectl exec -ti busybox -- nslookup kubernetes.default
  Server:    10.96.0.10
  Address 1: 10.96.0.10
  
  nslookup: can't resolve 'kubernetes.default'
  </code></pre>
  <p>cluster is running RHEL 8 with latest update</p>
  <p>followed this steps: <a href="https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/" rel="noreferrer">https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/</a></p>
  <p><strong>nslookup command not able to reach nameserver</strong></p>
  <pre><code># kubectl exec -i -t dnsutils -- nslookup kubernetes.default
  ;; connection timed out; no servers could be reached
  
  command terminated with exit code 1
  </code></pre>
  <p><strong>resolve.conf file</strong></p>
  <pre><code># kubectl exec -ti dnsutils -- cat /etc/resolv.conf
  search default.svc.cluster.local svc.cluster.local cluster.local 
  nameserver 10.96.0.10
  options ndots:5
  </code></pre>
  <p><strong>DNS pods running</strong></p>
  <pre><code># kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
  NAME                      READY   STATUS    RESTARTS   AGE
  coredns-74ff55c5b-472vx   1/1     Running   1          85m
  coredns-74ff55c5b-c75bq   1/1     Running   1          85m
  </code></pre>
  <p><strong>DNS pod logs</strong></p>
  <pre><code> kubectl logs --namespace=kube-system -l k8s-app=kube-dns
  .:53
  [INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
  CoreDNS-1.7.0
  linux/amd64, go1.14.4, f59c03d
  .:53
  [INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
  CoreDNS-1.7.0
  linux/amd64, go1.14.4, f59c03d
  </code></pre>
  <p><strong>Service is defined</strong></p>
  <pre><code># kubectl get svc --namespace=kube-system
  NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
  kube-dns   ClusterIP   10.96.0.10   &lt;none&gt;        53/UDP,53/TCP,9153/TCP   86m
  
  **I can see the endpoints of DNS pod**
  
  # kubectl get endpoints kube-dns --namespace=kube-system
  NAME       ENDPOINTS                                               AGE
  kube-dns   10.203.0.5:53,10.203.0.6:53,10.203.0.5:53 + 3 more...   86m
  </code></pre>
  <p><strong>enabled the logging, but didn't see traffic coming to DNS pod</strong></p>
  <pre><code># kubectl logs --namespace=kube-system -l k8s-app=kube-dns
  .:53
  [INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
  CoreDNS-1.7.0
  linux/amd64, go1.14.4, f59c03d
  .:53
  [INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
  CoreDNS-1.7.0
  linux/amd64, go1.14.4, f59c03d
  </code></pre>
  <p><strong>I can ping DNS POD</strong></p>
  <pre><code># kubectl exec -i -t dnsutils -- ping 10.203.0.5
  PING 10.203.0.5 (10.203.0.5): 56 data bytes
  64 bytes from 10.203.0.5: seq=0 ttl=62 time=6.024 ms
  64 bytes from 10.203.0.5: seq=1 ttl=62 time=6.052 ms
  64 bytes from 10.203.0.5: seq=2 ttl=62 time=6.175 ms
  64 bytes from 10.203.0.5: seq=3 ttl=62 time=6.000 ms
  ^C
  --- 10.203.0.5 ping statistics ---
  4 packets transmitted, 4 packets received, 0% packet loss
  round-trip min/avg/max = 6.000/6.062/6.175 ms
  </code></pre>
  <p><strong>nmap show port filtered</strong></p>
  <pre><code># ke netshoot-6f677d4fdf-5t5cb -- nmap 10.203.0.5
  Starting Nmap 7.80 ( https://nmap.org ) at 2021-01-15 22:29 UTC
  Nmap scan report for 10.203.0.5
  Host is up (0.0060s latency).
  Not shown: 997 closed ports
  PORT     STATE    SERVICE
  53/tcp   filtered domain
  8080/tcp filtered http-proxy
  8181/tcp filtered intermapper
  
  Nmap done: 1 IP address (1 host up) scanned in 14.33 seconds
  </code></pre>
  <p><strong>If I schedule the POD on master node, nslookup works nmap show port open?</strong></p>
  <pre><code># ke netshoot -- bash
  bash-5.0# nslookup kubernetes.default
  Server:         10.96.0.10
  Address:        10.96.0.10#53
  
  Name:   kubernetes.default.svc.cluster.local
  Address: 10.96.0.1
  
   nmap -p 53 10.96.0.10
  Starting Nmap 7.80 ( https://nmap.org ) at 2021-01-15 22:46 UTC
  Nmap scan report for kube-dns.kube-system.svc.cluster.local (10.96.0.10)
  Host is up (0.000098s latency).
  
  PORT   STATE SERVICE
  53/tcp open  domain
  
  Nmap done: 1 IP address (1 host up) scanned in 0.14 seconds
  </code></pre>
  <p>Why nslookup from POD running on worker node is not working? how to troubleshoot this issue?</p>
  <p>I re-build the server two times, still same issue.</p>
  <p>Thanks</p>
  <p>SR</p>
  <h2>Update adding kubeadm config file</h2>
  <pre><code># cat kubeadm-config.yaml
  ---
  apiVersion: kubeadm.k8s.io/v1beta2
  bootstrapTokens:
  - groups:
    - system:bootstrappers:kubeadm:default-node-token
    token: abcdef.0123456789abcdef
    ttl: 24h0m0s
    usages:
    - signing
    - authentication
  kind: InitConfiguration
  nodeRegistration:
    criSocket: unix:///run/containerd/containerd.sock
    taints:
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
    kubeletExtraArgs:
      cgroup-driver: &quot;systemd&quot;
  ---
  apiVersion: kubeadm.k8s.io/v1beta2
  kind: ClusterConfiguration
  kubernetesVersion: stable
  controlPlaneEndpoint: &quot;master01:6443&quot;
  networking:
    dnsDomain: cluster.local
    podSubnet: 10.0.0.0/14
    serviceSubnet: 10.96.0.0/12
  ---
  apiVersion: kubeproxy.config.k8s.io/v1alpha1
  kind: KubeProxyConfiguration
  mode: &quot;ipvs
  </code></pre>
  <p>&quot;</p>