title: k3s - networking between pods not working
body: <p>I'm struggling with this cross-communication between pods even though clusterIP services are set up for them. All the pods are on the same master node, and in the same namespace. In Summary:</p>
  <pre><code>$ kubectl get pods -o wide
  NAME                         READY   STATUS    RESTARTS   AGE    IP           NODE          NOMINATED NODE   READINESS GATES
  nginx-744f4df6df-rxhph       1/1     Running   0          136m   10.42.0.31   raspberrypi   &lt;none&gt;           &lt;none&gt;
  nginx-2-867f4f8859-csn48     1/1     Running   0          134m   10.42.0.32   raspberrypi   &lt;none&gt;           &lt;none&gt;
  
  $ kubectl get svc -o wide
  NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE    SELECTOR
  nginx-service    ClusterIP   10.43.155.201   &lt;none&gt;        80/TCP                       136m   app=nginx
  nginx-service2   ClusterIP   10.43.182.138   &lt;none&gt;        85/TCP                       134m   app=nginx-2
  </code></pre>
  <p>where I can't <em>curl http://nginx-service2:85</em> from within nginx container, or vice versa... while I validated that this worked from my docker desktop installation:</p>
  <pre><code># docker desktop
  root@nginx-7dc45fbd74-7prml:/# curl http://nginx-service2:85
  &lt;!DOCTYPE html&gt;
  &lt;html&gt;
  &lt;head&gt;
  &lt;title&gt;Welcome to nginx!&lt;/title&gt;
  &lt;style&gt;
      body {
          width: 35em;
          margin: 0 auto;
          font-family: Tahoma, Verdana, Arial, sans-serif;
      }
  &lt;/style&gt;
  &lt;/head&gt;
  &lt;body&gt;
  &lt;h1&gt;Welcome to nginx!&lt;/h1&gt;
  &lt;p&gt;If you see this page, the nginx web server is successfully installed and
  working. Further configuration is required.&lt;/p&gt;
  
  &lt;p&gt;For online documentation and support please refer to
  &lt;a href=&quot;http://nginx.org/&quot;&gt;nginx.org&lt;/a&gt;.&lt;br/&gt;
  Commercial support is available at
  &lt;a href=&quot;http://nginx.com/&quot;&gt;nginx.com&lt;/a&gt;.&lt;/p&gt;
  
  &lt;p&gt;&lt;em&gt;Thank you for using nginx.&lt;/em&gt;&lt;/p&gt;
  &lt;/body&gt;
  &lt;/html&gt;
  
  # k3s
  root@nginx-744f4df6df-rxhph:/# curl http://nginx-service2.pwk3spi-vraptor:85
  curl: (6) Could not resolve host: nginx-service2.pwk3spi-vraptor
  </code></pre>
  <p>After googling the issue (and please correct me if I'm wrong) it seems like a coredns issue, because looking at the logs and see the error timeouts:</p>
  <pre><code>$ kubectl get pods -n kube-system
  NAME                                     READY   STATUS      RESTARTS   AGE
  helm-install-traefik-qr2bd               0/1     Completed   0          153d
  metrics-server-7566d596c8-nnzg2          1/1     Running     59         148d
  svclb-traefik-kjbbr                      2/2     Running     60         153d
  traefik-758cd5fc85-wzjrn                 1/1     Running     20         62d
  local-path-provisioner-6d59f47c7-4hvf2   1/1     Running     72         148d
  coredns-7944c66d8d-gkdp4                 1/1     Running     0          3m47s
  
  $ kubectl logs coredns-7944c66d8d-gkdp4 -n kube-system
  .:53
  [INFO] plugin/reload: Running configuration MD5 = 1c648f07b77ab1530deca4234afe0d03
  CoreDNS-1.6.9
  linux/arm, go1.14.1, 1766568
  [ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:50482-&gt;192.168.8.109:53: i/o timeout
  [ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:34160-&gt;192.168.8.109:53: i/o timeout
  [ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:53485-&gt;192.168.8.109:53: i/o timeout
  [ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:46642-&gt;192.168.8.109:53: i/o timeout
  [ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:55329-&gt;192.168.8.109:53: i/o timeout
  [ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:44471-&gt;192.168.8.109:53: i/o timeout
  [ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:49182-&gt;192.168.8.109:53: i/o timeout
  [ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:54082-&gt;192.168.8.109:53: i/o timeout
  [ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:48151-&gt;192.168.8.109:53: i/o timeout
  [ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:48599-&gt;192.168.8.109:53: i/o timeout
  </code></pre>
  <p>where people recommended</p>
  <ul>
  <li>changing the coredns config map to forward to your master node IP</li>
  </ul>
  <blockquote>
  <p>... other CoreFile stuff</p>
  <p>forward . <em>host server IP</em></p>
  <p>... other CoreFile stuff</p>
  </blockquote>
  <ul>
  <li>or adding your coredns clusterip IP as a nameserver to /etc/resolve.conf</li>
  </ul>
  <blockquote>
  <p>search default.svc.cluster.local svc.cluster.local cluster.local</p>
  <p>nameserver 10.42.0.38</p>
  <p>nameserver 192.168.8.1</p>
  <p>nameserver fe80::266:19ff:fea7:85e7%wlan0</p>
  </blockquote>
  <p>, however didn't find that these solutions worked.</p>
  <p>Details for reference:</p>
  <pre><code>$ kubectl get nodes -o wide
  NAME          STATUS   ROLES    AGE    VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
  raspberrypi   Ready    master   153d   v1.18.9+k3s1   192.168.8.109   &lt;none&gt;        Raspbian GNU/Linux 10 (buster)   5.10.9-v7l+      containerd://1.3.3-k3s2
  
  $ kubectl get svc -n kube-system -o wide
  NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE    SELECTOR
  kube-dns             ClusterIP      10.43.0.10      &lt;none&gt;          53/UDP,53/TCP,9153/TCP       153d   k8s-app=kube-dns
  metrics-server       ClusterIP      10.43.205.8     &lt;none&gt;          443/TCP                      153d   k8s-app=metrics-server
  traefik-prometheus   ClusterIP      10.43.222.138   &lt;none&gt;          9100/TCP                     153d   app=traefik,release=traefik
  traefik              LoadBalancer   10.43.249.133   192.168.8.109   80:31222/TCP,443:32509/TCP   153d   app=traefik,release=traefik
  
  $ kubectl get ep kube-dns -n kube-system
  NAME       ENDPOINTS                                     AGE
  kube-dns   10.42.0.38:53,10.42.0.38:9153,10.42.0.38:53   153d
  </code></pre>
  <p>No idea where I'm going wrong, or if I focused on the wrong stuff, or how to continue. Any help will be much appreciated, please.</p>