Answers: <p>I managed to solve the issue by changing the configuration of <code>nodelocaldns</code>:</p>
  <pre><code>kubectl edit configmap -n kube-system nodelocaldns
  </code></pre>
  <p>change from</p>
  <pre><code>cluster.local:53 {
      errors
      cache {
          success 9984 30
          denial 9984 5
      }
      reload
      loop
      bind 169.254.25.10
      forward . 10.233.0.3 {
          force_tcp
      }
      prometheus :9253
      health 169.254.25.10:9254
  }
  in-addr.arpa:53 {
      errors
      cache 30
      reload
      loop
      bind 169.254.25.10
      forward . 10.233.0.3 {
          force_tcp
      }
      prometheus :9253
  }
  ip6.arpa:53 {
      errors
      cache 30
      reload
      loop
      bind 169.254.25.10
      forward . 10.233.0.3 {
          force_tcp
      }
      prometheus :9253
  }
  .:53 {
      errors
      cache 30
      reload
      loop
      bind 169.254.25.10
      forward . /etc/resolv.conf
      prometheus :9253
  }
  </code></pre>
  <p>to this:</p>
  <pre><code>cluster.local:53 {
      errors
      cache {
          success 9984 30
          denial 9984 5
      }
      reload
      loop
      bind 169.254.25.10
      forward . 10.233.0.3 {
          force_tcp
      }
      prometheus :9253
      health 169.254.25.10:9254
  }
  in-addr.arpa:53 {
      errors
      cache 30
      reload
      loop
      bind 169.254.25.10
      forward . 10.233.0.3 {
          force_tcp
      }
      prometheus :9253
  }
  ip6.arpa:53 {
      errors
      cache 30
      reload
      loop
      bind 169.254.25.10
      forward . 10.233.0.3 {
          force_tcp
      }
      prometheus :9253
  }
  .:53 {
      errors
      cache 30
      reload
      loop
      bind 169.254.25.10
      forward . 10.233.0.3 {
          force_tcp
      }
      prometheus :9253
  }
  </code></pre>
  <p><em>look at the last block</em></p>
  <p>I am curious what caused this issue in the first place?</p>
  
Answers: CC BY-SA 4.0
Answers: 2024-08-19T11:05:37.420Z
Answers: 2024-08-19T11:05:37.420Z
Answers: badger
Answers: <p>In Kubernetes there is a convention that the cluster DNS service is on the <code>.10</code> IP address.</p>
  <p>So it's expected that you'll configure the ClusterIP for CoreDNS to request that particular IP.</p>
  <p>If not, you'll need to configure kubelet on each node to change what it injects into the <code>/etc/resolv.conf</code> file of most pods.</p>
  
Answers: CC BY-SA 4.0
Answers: 2024-08-19T20:42:31.023Z
Answers: Slightly more detail: https://serverfault.com/a/1158944/431900
Answers: benjimin
Answers: CC BY-SA 4.0
Answers: 2024-08-19T20:23:02.783Z
Answers: 2024-08-19T20:28:53.897Z
Answers: 2024-08-19T20:28:53.897Z
Answers: benjimin
Body: <p>I setup a Kubernetes cluster using Kubespray. it works flawlessly until recently I encountered something strange.</p>
  <p>I created a BIND DNS deployment using the following manifest.</p>
  <pre><code>apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: bind
    labels:
      app: bind
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: bind
    template:
      metadata:
        labels:
          app: bind
      spec:
        containers:
        - name: bind
          image: sameersbn/bind:9.16.1-20200524
          ports:
          - containerPort: 53
            name: tcp
            protocol: TCP 
          - containerPort: 53
            name: udp
            protocol: UDP
          - containerPort: 10000
            name: web
            protocol: TCP
          volumeMounts:
          - name: bind-data
            mountPath: /data
        volumes:
        - name: bind-data
          hostPath:
            path: /data
  ---
  apiVersion: v1
  kind: Service
  metadata:
    name: bind
  spec:
    type: ClusterIP
    ports:
    - port: 53
      name: tcp
      targetPort: tcp
      protocol: TCP
    - port: 53
      name: udp
      targetPort: udp
      protocol: UDP
    - port: 10000
      name: web
      targetPort: web
      protocol: TCP
    selector:
      app: bind
  </code></pre>
  <p>Then I've defined a master zone and created a forward dns record in it.</p>
  <pre><code>test.arman.local -&gt; 192.168.230.216 
  </code></pre>
  <p>Next I have configured the configmap of coredns to use it:</p>
  <pre><code>kubectl edit configmap -n kube-system coredns
  
  apiVersion: v1
  data:
    Corefile: |
      .:53 {
          errors
          health {
              lameduck 5s
          }
          ready
          kubernetes cluster.local in-addr.arpa ip6.arpa {
              pods insecure
              fallthrough in-addr.arpa ip6.arpa
              ttl 30
          }
          prometheus :9153
          forward . /etc/resolv.conf
          cache 30
          loop
          reload
          loadbalance
      }
      arman.local:53 {
          errors
          cache 30
          forward . 10.233.28.178
      }
  kind: ConfigMap
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {&quot;apiVersion&quot;:&quot;v1&quot;,&quot;data&quot;:{&quot;Corefile&quot;:&quot;.:53 {\n    errors {\n    }\n    health {\n        lameduck 5s\n    }\n    ready\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n      pods insecure\n      fallthrough in-addr.arpa ip6.arpa\n    }\n    prometheus :9153\n    forward . /etc/resolv.conf {\n      prefer_udp\n      max_concurrent 1000\n    }\n    cache 30\n\n    loop\n    reload\n    loadbalance\n}\n&quot;},&quot;kind&quot;:&quot;ConfigMap&quot;,&quot;metadata&quot;:{&quot;annotations&quot;:{},&quot;labels&quot;:{&quot;addonmanager.kubernetes.io/mode&quot;:&quot;EnsureExists&quot;},&quot;name&quot;:&quot;coredns&quot;,&quot;namespace&quot;:&quot;kube-system&quot;}}
    creationTimestamp: &quot;2024-08-08T10:26:32Z&quot;
    labels:
      addonmanager.kubernetes.io/mode: EnsureExists
    name: coredns
    namespace: kube-system
    resourceVersion: &quot;2135167&quot;
    uid: 5457c863-3a34-4490-badb-036f4f0e1739
  </code></pre>
  <p>The service of BIND deployment is like:</p>
  <pre><code>kubectl get svc                 
  NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
  bind                          ClusterIP   10.233.28.178   &lt;none&gt;        53/TCP,53/UDP,10000/TCP      44h
  </code></pre>
  <p><em>The problem is it can't resolve <code>test.arman.local</code></em></p>
  <p>I've started to debug the problem I launched a <code>busybox</code> and used <code>nslookup</code></p>
  <pre><code>kubectl run -it busybox --image=busybox:1.28 --restart=Never -- sh
  </code></pre>
  <p>Here is the content of <code>/etc/resolv.conf</code> of busybox</p>
  <pre><code>/ # cat /etc/resolv.conf 
  search default.svc.cluster.local svc.cluster.local cluster.local
  nameserver 169.254.25.10
  options ndots:5
  </code></pre>
  <p><strong>I don't know what IP <code>169.254.25.10</code> is for because the service of dnsCore has different IP</strong>:</p>
  <pre><code>kubectl get svc -n kube-system 
  NAME      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
  coredns   ClusterIP   10.233.0.3   &lt;none&gt;        53/UDP,53/TCP,9153/TCP   10d
  </code></pre>
  <p>In the busybox I can successfully resolve the <code>test.arman.local</code> using the following command:</p>
  <pre><code>nslookup test.arman.local 10.233.28.178
  Server:    10.233.28.178
  Address 1: 10.233.28.178
  
  Name:      test.arman.local
  Address 1: 192.168.230.216
  / # ^C
  / # cat /etc/resolv.conf 
  search default.svc.cluster.local svc.cluster.local cluster.local
  nameserver 169.254.25.10
  options ndots:5
  </code></pre>
  <p>Or even use this:</p>
  <pre><code>nslookup test.arman.local 10.233.0.3 # This is for coredns
  Server:    10.233.0.3
  Address 1: 10.233.0.3 coredns.kube-system.svc.cluster.local
  
  Name:      test.arman.local
  Address 1: 192.168.230.216
  </code></pre>
  <p>We are using DNS/calico, for the sake of completeness I post the pods in kube-system:</p>
  <pre><code> kubectl get pods -n kube-system 
  NAME                                      READY   STATUS    RESTARTS        AGE
  calico-kube-controllers-c7cc688f8-rb59q   1/1     Running   1 (7d21h ago)   10d
  calico-node-5dglq                         1/1     Running   0               10d
  calico-node-c592m                         1/1     Running   0               10d
  calico-node-hbfqj                         1/1     Running   0               10d
  calico-node-s6jrl                         1/1     Running   0               10d
  coredns-857f949bf7-bprlf                  1/1     Running   0               65m
  coredns-857f949bf7-r9p87                  1/1     Running   0               65m
  dns-autoscaler-6ffb84bd6-fgdtz            1/1     Running   0               10d
  kube-apiserver-node1                      1/1     Running   1               10d
  kube-apiserver-node2                      1/1     Running   1               10d
  kube-controller-manager-node1             1/1     Running   4 (7d21h ago)   10d
  kube-controller-manager-node2             1/1     Running   7 (6d19h ago)   10d
  kube-proxy-hb2wn                          1/1     Running   0               10d
  kube-proxy-j5pfg                          1/1     Running   0               10d
  kube-proxy-qjj4n                          1/1     Running   0               10d
  kube-proxy-rp9mt                          1/1     Running   0               10d
  kube-scheduler-node1                      1/1     Running   4 (6d19h ago)   10d
  kube-scheduler-node2                      1/1     Running   6 (6d19h ago)   10d
  nginx-proxy-node3                         1/1     Running   0               10d
  nginx-proxy-node4                         1/1     Running   0               10d
  nodelocaldns-6sztv                        1/1     Running   0               10d
  nodelocaldns-bpt7s                        1/1     Running   0               10d
  nodelocaldns-n4rsk                        1/1     Running   0               10d
  nodelocaldns-n8q9w                        1/1     Running   0               10d
  </code></pre>
  <p>UPDATE: I found out it has something to do with <code>nodelocaldns</code> but I can not resolve the issue</p>
  
Comments: CC BY-SA 4.0
Comments: 2024-08-19T20:25:03.547Z
Comments: Probable duplicate of e.g. https://serverfault.com/questions/1009907/kubernetes-pod-etc-resolv-conf-has-the-wrong-nameserver
Comments: benjimin
Tags: <kubernetes>
Title: CoreDNS is not able to resolve names
UserDisplayName: badger
ThreadText: # CoreDNS is not able to resolve names
  
  (Asked by: username_0 on 2024-08-19)
  
  I setup a Kubernetes cluster using Kubespray. it works flawlessly until recently I encountered something strange.
  
  I created a BIND DNS deployment using the following manifest.
  
  ```
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: bind
    labels:
      app: bind
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: bind
    template:
      metadata:
        labels:
          app: bind
      spec:
        containers:
        - name: bind
          image: sameersbn/bind:9.16.1-20200524
          ports:
          - containerPort: 53
            name: tcp
            protocol: TCP 
          - containerPort: 53
            name: udp
            protocol: UDP
          - containerPort: 10000
            name: web
            protocol: TCP
          volumeMounts:
          - name: bind-data
            mountPath: /data
        volumes:
        - name: bind-data
          hostPath:
            path: /data
  ---
  apiVersion: v1
  kind: Service
  metadata:
    name: bind
  spec:
    type: ClusterIP
    ports:
    - port: 53
      name: tcp
      targetPort: tcp
      protocol: TCP
    - port: 53
      name: udp
      targetPort: udp
      protocol: UDP
    - port: 10000
      name: web
      targetPort: web
      protocol: TCP
    selector:
      app: bind
  
  ```
  
  Then I've defined a master zone and created a forward dns record in it.
  
  ```
  test.arman.local -> 192.168.230.216 
  
  ```
  
  Next I have configured the configmap of coredns to use it:
  
  ```
  kubectl edit configmap -n kube-system coredns
  
  apiVersion: v1
  data:
    Corefile: |
      .:53 {
          errors
          health {
              lameduck 5s
          }
          ready
          kubernetes cluster.local in-addr.arpa ip6.arpa {
              pods insecure
              fallthrough in-addr.arpa ip6.arpa
              ttl 30
          }
          prometheus :9153
          forward . /etc/resolv.conf
          cache 30
          loop
          reload
          loadbalance
      }
      arman.local:53 {
          errors
          cache 30
          forward . 10.233.28.178
      }
  kind: ConfigMap
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"v1","data":{"Corefile":".:53 {\n    errors {\n    }\n    health {\n        lameduck 5s\n    }\n    ready\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n      pods insecure\n      fallthrough in-addr.arpa ip6.arpa\n    }\n    prometheus :9153\n    forward . /etc/resolv.conf {\n      prefer_udp\n      max_concurrent 1000\n    }\n    cache 30\n\n    loop\n    reload\n    loadbalance\n}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"EnsureExists"},"name":"coredns","namespace":"kube-system"}}
    creationTimestamp: "2024-08-08T10:26:32Z"
    labels:
      addonmanager.kubernetes.io/mode: EnsureExists
    name: coredns
    namespace: kube-system
    resourceVersion: "2135167"
    uid: 5457c863-3a34-4490-badb-036f4f0e1739
  
  ```
  
  The service of BIND deployment is like:
  
  ```
  kubectl get svc                 
  NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
  bind                          ClusterIP   10.233.28.178   <none>        53/TCP,53/UDP,10000/TCP      44h
  
  ```
  
  *The problem is it can't resolve `test.arman.local`*
  
  I've started to debug the problem I launched a `busybox` and used `nslookup`
  
  ```
  kubectl run -it busybox --image=busybox:1.28 --restart=Never -- sh
  
  ```
  
  Here is the content of `/etc/resolv.conf` of busybox
  
  ```
  / # cat /etc/resolv.conf 
  search default.svc.cluster.local svc.cluster.local cluster.local
  nameserver 169.254.25.10
  options ndots:5
  
  ```
  
  **I don't know what IP `169.254.25.10` is for because the service of dnsCore has different IP**:
  
  ```
  kubectl get svc -n kube-system 
  NAME      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
  coredns   ClusterIP   10.233.0.3   <none>        53/UDP,53/TCP,9153/TCP   10d
  
  ```
  
  In the busybox I can successfully resolve the `test.arman.local` using the following command:
  
  ```
  nslookup test.arman.local 10.233.28.178
  Server:    10.233.28.178
  Address 1: 10.233.28.178
  
  Name:      test.arman.local
  Address 1: 192.168.230.216
  / # ^C
  / # cat /etc/resolv.conf 
  search default.svc.cluster.local svc.cluster.local cluster.local
  nameserver 169.254.25.10
  options ndots:5
  
  ```
  
  Or even use this:
  
  ```
  nslookup test.arman.local 10.233.0.3 # This is for coredns
  Server:    10.233.0.3
  Address 1: 10.233.0.3 coredns.kube-system.svc.cluster.local
  
  Name:      test.arman.local
  Address 1: 192.168.230.216
  
  ```
  
  We are using DNS/calico, for the sake of completeness I post the pods in kube-system:
  
  ```
   kubectl get pods -n kube-system 
  NAME                                      READY   STATUS    RESTARTS        AGE
  calico-kube-controllers-c7cc688f8-rb59q   1/1     Running   1 (7d21h ago)   10d
  calico-node-5dglq                         1/1     Running   0               10d
  calico-node-c592m                         1/1     Running   0               10d
  calico-node-hbfqj                         1/1     Running   0               10d
  calico-node-s6jrl                         1/1     Running   0               10d
  coredns-857f949bf7-bprlf                  1/1     Running   0               65m
  coredns-857f949bf7-r9p87                  1/1     Running   0               65m
  dns-autoscaler-6ffb84bd6-fgdtz            1/1     Running   0               10d
  kube-apiserver-node1                      1/1     Running   1               10d
  kube-apiserver-node2                      1/1     Running   1               10d
  kube-controller-manager-node1             1/1     Running   4 (7d21h ago)   10d
  kube-controller-manager-node2             1/1     Running   7 (6d19h ago)   10d
  kube-proxy-hb2wn                          1/1     Running   0               10d
  kube-proxy-j5pfg                          1/1     Running   0               10d
  kube-proxy-qjj4n                          1/1     Running   0               10d
  kube-proxy-rp9mt                          1/1     Running   0               10d
  kube-scheduler-node1                      1/1     Running   4 (6d19h ago)   10d
  kube-scheduler-node2                      1/1     Running   6 (6d19h ago)   10d
  nginx-proxy-node3                         1/1     Running   0               10d
  nginx-proxy-node4                         1/1     Running   0               10d
  nodelocaldns-6sztv                        1/1     Running   0               10d
  nodelocaldns-bpt7s                        1/1     Running   0               10d
  nodelocaldns-n4rsk                        1/1     Running   0               10d
  nodelocaldns-n8q9w                        1/1     Running   0               10d
  
  ```
  
  UPDATE: I found out it has something to do with `nodelocaldns` but I can not resolve the issue
  
  **Tags:** kubernetes
  
  > **Comments:**
  > **username_1**: Probable duplicate of e.g. https://serverfault.com/questions/1009907/kubernetes-pod-etc-resolv-conf-has-the-wrong-nameserver  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2024-08-19. Score: 0 | Accepted answer
  
  I managed to solve the issue by changing the configuration of `nodelocaldns`:
  
  ```
  kubectl edit configmap -n kube-system nodelocaldns
  
  ```
  
  change from
  
  ```
  cluster.local:53 {
      errors
      cache {
          success 9984 30
          denial 9984 5
      }
      reload
      loop
      bind 169.254.25.10
      forward . 10.233.0.3 {
          force_tcp
      }
      prometheus :9253
      health 169.254.25.10:9254
  }
  in-addr.arpa:53 {
      errors
      cache 30
      reload
      loop
      bind 169.254.25.10
      forward . 10.233.0.3 {
          force_tcp
      }
      prometheus :9253
  }
  ip6.arpa:53 {
      errors
      cache 30
      reload
      loop
      bind 169.254.25.10
      forward . 10.233.0.3 {
          force_tcp
      }
      prometheus :9253
  }
  .:53 {
      errors
      cache 30
      reload
      loop
      bind 169.254.25.10
      forward . /etc/resolv.conf
      prometheus :9253
  }
  
  ```
  
  to this:
  
  ```
  cluster.local:53 {
      errors
      cache {
          success 9984 30
          denial 9984 5
      }
      reload
      loop
      bind 169.254.25.10
      forward . 10.233.0.3 {
          force_tcp
      }
      prometheus :9253
      health 169.254.25.10:9254
  }
  in-addr.arpa:53 {
      errors
      cache 30
      reload
      loop
      bind 169.254.25.10
      forward . 10.233.0.3 {
          force_tcp
      }
      prometheus :9253
  }
  ip6.arpa:53 {
      errors
      cache 30
      reload
      loop
      bind 169.254.25.10
      forward . 10.233.0.3 {
          force_tcp
      }
      prometheus :9253
  }
  .:53 {
      errors
      cache 30
      reload
      loop
      bind 169.254.25.10
      forward . 10.233.0.3 {
          force_tcp
      }
      prometheus :9253
  }
  
  ```
  
  *look at the last block*
  
  I am curious what caused this issue in the first place?
  
  
  ---
  
  ### Answer by username_1 on 2024-08-19. Score: 0
  
  In Kubernetes there is a convention that the cluster DNS service is on the `.10` IP address.
  
  So it's expected that you'll configure the ClusterIP for CoreDNS to request that particular IP.
  
  If not, you'll need to configure kubelet on each node to change what it injects into the `/etc/resolv.conf` file of most pods.
  
  > **Comments:**
  > **username_1**: Slightly more detail: https://serverfault.com/a/1158944/431900  
  
  ---
  
  