Answers: <p>So after deleting all canal PODs and letting them recreate, everything started to work again as it should.
  Seems like the recreation of those PODs somehow recreated routes and added the Worker node. Not sure why it didnt work right after a worker reboot, later when removing and re-joining this worker to the cluster, it got assigned a new CIDR, after canal POD recreation it started to work.</p>
  
Answers: CC BY-SA 4.0
Answers: 2023-12-20T14:53:04.147Z
Answers: 2023-12-20T14:53:04.147Z
Answers: Ghorio Fuitton
Answers: <p>Do you have vxlan in canal.yaml? Then change that to host-gw and see if it fixes the problem. flannel will communicate new routes to the kernel with netlink sockets then.</p>
  <pre><code>  net-conf.json: |
      {
        &quot;Network&quot;: &quot;10.244.0.0/16&quot;,
        &quot;Backend&quot;: {
          &quot;Type&quot;: &quot;vxlan&quot; # CHANGE TO host-gw
        }
      }
  </code></pre>
  <p>More info here.</p>
  <p><a href="https://devops.stackexchange.com/questions/18555/kubernetes-access-nodeport-fails-master-cannot-reach-workers/18557#18557">kubernetes access nodeport fails, master cannot reach workers</a></p>
  <p>and on the flannel github.</p>
  <p><a href="https://github.com/flannel-io/flannel" rel="nofollow noreferrer">https://github.com/flannel-io/flannel</a></p>
  <p>If it still doesn't work, try adding the specific network interface flannel must work on. If you dont it will claim the first interface it finds.</p>
  <pre><code> - name: kube-flannel
            image: quay.io/coreos/flannel:v0.9.1
            command: [ &quot;/opt/bin/flanneld&quot;, &quot;--ip-masq&quot;, &quot;--kube-subnet-mgr&quot; ]
  </code></pre>
  <p>add --iface=iface_you_want to &quot;command&quot;</p>
  
Answers: CC BY-SA 4.0
Answers: 2023-12-14T09:57:31.673Z
Answers: 2023-12-14T09:57:31.673Z
Answers: Serve Laurijssen
Body: <p>Really looking for some advice and help, I have run out of ideas. Basically everything was working fine, 1 master and 3 workers. One Worker node was turned off to increase CPU and RAM resources and once turned back on, PODs running on this worker were not able to reach any other PODs and also DNS resolving was not working, so no external network connection also was working, e.g. google.com etc.
  Here is a list of POD cidrs</p>
  <pre><code>$ kubectl get nodes -o json | jq -r '.items[].spec.podCIDR' | sort
  Master - 10.244.0.0/24
  Worker1 - 10.244.1.0/24
  Worker2 - 10.244.2.0/24
  Worker3 - 10.244.5.0/24
  </code></pre>
  <p>The one which was restarted and stopped working is <strong>Worker3</strong>. When running dnsutils container on this node and trying some debuging</p>
  <pre><code>root@dnsutils:/# nslookup kubernetes.default
  ;; connection timed out; no servers could be reached
  </code></pre>
  <p>Firewalld has been disabled and turned off.
  Also when trying to ping all the networks from Worker3, only 10.244.1.0 and 10.244.5.0 (itself) is pinging</p>
  <pre><code>root@dnsutils:/# ping 10.244.0.0
  PING 10.244.0.0 (10.244.0.0) 56(84) bytes of data.
  ^C
  --- 10.244.0.0 ping statistics ---
  2 packets transmitted, 0 received, 100% packet loss, time 1043ms
  
  root@dnsutils:/# ping 10.244.1.0
  PING 10.244.1.0 (10.244.1.0) 56(84) bytes of data.
  64 bytes from 10.244.1.0: icmp_seq=1 ttl=63 time=0.721 ms
  64 bytes from 10.244.1.0: icmp_seq=2 ttl=63 time=0.947 ms
  ^C
  --- 10.244.1.0 ping statistics ---
  2 packets transmitted, 2 received, 0% packet loss, time 1041ms
  rtt min/avg/max/mdev = 0.721/0.834/0.947/0.113 ms
  root@dnsutils:/# ping 10.244.2.0
  PING 10.244.2.0 (10.244.2.0) 56(84) bytes of data.
  ^C
  --- 10.244.2.0 ping statistics ---
  2 packets transmitted, 0 received, 100% packet loss, time 1024ms
  
  root@dnsutils:/# ping 10.244.5.0
  PING 10.244.5.0 (10.244.5.0) 56(84) bytes of data.
  64 bytes from 10.244.5.0: icmp_seq=1 ttl=64 time=0.116 ms
  64 bytes from 10.244.5.0: icmp_seq=2 ttl=64 time=0.142 ms
  ^C
  --- 10.244.5.0 ping statistics ---
  2 packets transmitted, 2 received, 0% packet loss, time 1038ms
  rtt min/avg/max/mdev = 0.116/0.129/0.142/0.013 ms
  </code></pre>
  <p>I believe the issue is because it cant reach coredns service/pods, which both are currently on 10.244.0.35 and 10.244.0.36 IPs</p>
  <pre><code>NAME                                                                 READY   STATUS    RESTARTS         AGE     IP              NODE        NOMINATED NODE   READINESS GATES
  calico-kube-controllers-7b564c7d85-rkc98                             1/1     Running   0                278d    10.244.0.30     Master       &lt;none&gt;           &lt;none&gt;
  canal-5cffv                                                          2/2     Running   4 (378d ago)     630d    10.203.8.167    Worker3      &lt;none&gt;           &lt;none&gt;
  canal-p822m                                                          2/2     Running   2 (474d ago)     630d    10.203.8.165    Master       &lt;none&gt;           &lt;none&gt;
  canal-scdrs                                                          2/2     Running   4 (351d ago)     630d    10.203.40.166   Worker2      &lt;none&gt;           &lt;none&gt;
  canal-xkx7f                                                          2/2     Running   0                5h16m   10.203.8.174    Worker4      &lt;none&gt;           &lt;none&gt;
  coredns-cc4c6799f-4tjbd                                              1/1     Running   0                11h     10.244.0.36     Master       &lt;none&gt;           &lt;none&gt;
  coredns-cc4c6799f-bkq8n                                              1/1     Running   0                11h     10.244.0.35     Master       &lt;none&gt;           &lt;none&gt;
  etcd-Master                                                          1/1     Running   6 (474d ago)     630d    10.203.8.165    Master       &lt;none&gt;           &lt;none&gt;
  kube-apiserver-Master                                                1/1     Running   6 (474d ago)     630d    10.203.8.165    Master       &lt;none&gt;           &lt;none&gt;
  kube-controller-manager-Master                                       1/1     Running   10 (265d ago)    392d    10.203.8.165    Master       &lt;none&gt;           &lt;none&gt;
  kube-proxy-9n4wc                                                     1/1     Running   1 (474d ago)     630d    10.203.8.165    Master       &lt;none&gt;           &lt;none&gt;
  kube-proxy-rtccq                                                     1/1     Running   2 (351d ago)     630d    10.203.40.166   Worker2      &lt;none&gt;           &lt;none&gt;
  kube-proxy-vtr55                                                     1/1     Running   0                5h16m   10.203.8.174    Worker4      &lt;none&gt;           &lt;none&gt;
  kube-proxy-w9wkk                                                     1/1     Running   2 (378d ago)     630d    10.203.8.167    Worker3      &lt;none&gt;           &lt;none&gt;
  kube-scheduler-Master                                                1/1     Running   326 (265d ago)   630d    10.203.8.165    Master       &lt;none&gt;           &lt;none&gt;
  kube-state-metrics-8945855d-pvqqt                                    1/1     Running   0                139d    10.244.2.89     Worker3      &lt;none&gt;           &lt;none&gt;
  metrics-server-7cf8b65d65-qpnzm                                      1/1     Running   2 (6h43m ago)    139d    10.244.2.83     Worker3      &lt;none&gt;           &lt;none&gt;
  </code></pre>
  <p>Running dnsutils on Worker2 for example</p>
  <pre><code>root@dnsutils2:/# nslookup kubernetes.default
  Server:     10.96.0.10
  Address:    10.96.0.10#53
  
  Name:   kubernetes.default.svc.cluster.local
  Address: 10.96.0.1
  
  root@dnsutils2:/# ping 10.244.0.0
  PING 10.244.0.0 (10.244.0.0) 56(84) bytes of data.
  64 bytes from 10.244.0.0: icmp_seq=1 ttl=63 time=0.787 ms
  64 bytes from 10.244.0.0: icmp_seq=2 ttl=63 time=0.835 ms
  ^C
  --- 10.244.0.0 ping statistics ---
  2 packets transmitted, 2 received, 0% packet loss, time 1061ms
  rtt min/avg/max/mdev = 0.787/0.811/0.835/0.024 ms
  root@dnsutils2:/# ping 10.244.1.0
  PING 10.244.1.0 (10.244.1.0) 56(84) bytes of data.
  64 bytes from 10.244.1.0: icmp_seq=1 ttl=64 time=0.129 ms
  64 bytes from 10.244.1.0: icmp_seq=2 ttl=64 time=0.152 ms
  ^C
  --- 10.244.1.0 ping statistics ---
  2 packets transmitted, 2 received, 0% packet loss, time 1021ms
  rtt min/avg/max/mdev = 0.129/0.140/0.152/0.016 ms
  root@dnsutils2:/# ping 10.244.2.0
  PING 10.244.2.0 (10.244.2.0) 56(84) bytes of data.
  64 bytes from 10.244.2.0: icmp_seq=1 ttl=63 time=1.03 ms
  64 bytes from 10.244.2.0: icmp_seq=2 ttl=63 time=0.933 ms
  ^C
  --- 10.244.2.0 ping statistics ---
  2 packets transmitted, 2 received, 0% packet loss, time 1001ms
  rtt min/avg/max/mdev = 0.933/0.984/1.036/0.060 ms
  root@dnsutils2:/# ping 10.244.5.0
  PING 10.244.5.0 (10.244.5.0) 56(84) bytes of data.
  64 bytes from 10.244.5.0: icmp_seq=1 ttl=63 time=0.787 ms
  64 bytes from 10.244.5.0: icmp_seq=2 ttl=63 time=0.995 ms
  ^C
  --- 10.244.5.0 ping statistics ---
  2 packets transmitted, 2 received, 0% packet loss, time 1024ms
  rtt min/avg/max/mdev = 0.787/0.891/0.995/0.104 ms
  </code></pre>
  <p>Cluster was initialized running below command</p>
  <pre><code>kubeadm init --apiserver-advertise-address=10.203.8.165 --pod-network-cidr=10.244.0.0/16
  </code></pre>
  <p>And for network cni plugin flannel-canal was used</p>
  <pre><code>kubectl apply -f https://projectcalico.docs.tigera.io/manifests/canal.yaml
  </code></pre>
  <p>I would really appreciate any help on where to start debugging this.
  Basically this Worker3 was working just fine before a restart and admins say that nothing was changed from network side or security groups etc. This is running on premises on VMware VMs.</p>
  <p>I have also tried <code>kubeadm reset</code> and then <code>iptables -F &amp;&amp; iptables -t nat -F &amp;&amp; iptables -t mangle -F &amp;&amp; iptables -X</code> , rebooting the node and then <code>kubeadm join</code> but still the issue persists. No idea why this specific Worker3 wont ping anything besides 10.244.1.0 and 10.244.5.0 (itself)</p>
  <p>I am confused about why wouldnt node be able to ping 10.244.2.0 network for example, but is able to ping 10.244.1.0 although there is iptables</p>
  <pre><code>-A FORWARD -s 10.244.0.0/16 -j ACCEPT
  -A FORWARD -d 10.244.0.0/16 -j ACCEPT
  </code></pre>
  <p>and also routes</p>
  <pre><code># route -n
  Kernel IP routing table
  Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
  0.0.0.0         10.203.8.254    0.0.0.0         UG    100    0        0 ens192
  10.203.8.0      0.0.0.0         255.255.255.0   U     100    0        0 ens192
  10.244.0.0      10.244.0.0      255.255.255.0   UG    0      0        0 flannel.1
  10.244.1.0      10.244.1.0      255.255.255.0   UG    0      0        0 flannel.1
  10.244.2.0      10.244.2.0      255.255.255.0   UG    0      0        0 flannel.1
  10.244.5.2      0.0.0.0         255.255.255.255 UH    0      0        0 cali952544557fa
  10.244.5.3      0.0.0.0         255.255.255.255 UH    0      0        0 calic15b507754a
  10.244.5.4      0.0.0.0         255.255.255.255 UH    0      0        0 calif987de568c4
  10.244.5.7      0.0.0.0         255.255.255.255 UH    0      0        0 cali969530ca43d
  10.244.5.8      0.0.0.0         255.255.255.255 UH    0      0        0 calib3c61c3cba9
  </code></pre>
  
Tags: <kubernetes>
Title: Kubernetes worker node not able to reach POD network CIDR after reboot
UserDisplayName: Ghorio Fuitton
ThreadText: # Kubernetes worker node not able to reach POD network CIDR after reboot
  
  (Asked by: username_0 on 2023-12-09)
  
  Really looking for some advice and help, I have run out of ideas. Basically everything was working fine, 1 master and 3 workers. One Worker node was turned off to increase CPU and RAM resources and once turned back on, PODs running on this worker were not able to reach any other PODs and also DNS resolving was not working, so no external network connection also was working, e.g. google.com etc.
  Here is a list of POD cidrs
  
  ```
  $ kubectl get nodes -o json | jq -r '.items[].spec.podCIDR' | sort
  Master - 10.244.0.0/24
  Worker1 - 10.244.1.0/24
  Worker2 - 10.244.2.0/24
  Worker3 - 10.244.5.0/24
  
  ```
  
  The one which was restarted and stopped working is **Worker3**. When running dnsutils container on this node and trying some debuging
  
  ```
  root@dnsutils:/# nslookup kubernetes.default
  ;; connection timed out; no servers could be reached
  
  ```
  
  Firewalld has been disabled and turned off.
  Also when trying to ping all the networks from Worker3, only 10.244.1.0 and 10.244.5.0 (itself) is pinging
  
  ```
  root@dnsutils:/# ping 10.244.0.0
  PING 10.244.0.0 (10.244.0.0) 56(84) bytes of data.
  ^C
  --- 10.244.0.0 ping statistics ---
  2 packets transmitted, 0 received, 100% packet loss, time 1043ms
  
  root@dnsutils:/# ping 10.244.1.0
  PING 10.244.1.0 (10.244.1.0) 56(84) bytes of data.
  64 bytes from 10.244.1.0: icmp_seq=1 ttl=63 time=0.721 ms
  64 bytes from 10.244.1.0: icmp_seq=2 ttl=63 time=0.947 ms
  ^C
  --- 10.244.1.0 ping statistics ---
  2 packets transmitted, 2 received, 0% packet loss, time 1041ms
  rtt min/avg/max/mdev = 0.721/0.834/0.947/0.113 ms
  root@dnsutils:/# ping 10.244.2.0
  PING 10.244.2.0 (10.244.2.0) 56(84) bytes of data.
  ^C
  --- 10.244.2.0 ping statistics ---
  2 packets transmitted, 0 received, 100% packet loss, time 1024ms
  
  root@dnsutils:/# ping 10.244.5.0
  PING 10.244.5.0 (10.244.5.0) 56(84) bytes of data.
  64 bytes from 10.244.5.0: icmp_seq=1 ttl=64 time=0.116 ms
  64 bytes from 10.244.5.0: icmp_seq=2 ttl=64 time=0.142 ms
  ^C
  --- 10.244.5.0 ping statistics ---
  2 packets transmitted, 2 received, 0% packet loss, time 1038ms
  rtt min/avg/max/mdev = 0.116/0.129/0.142/0.013 ms
  
  ```
  
  I believe the issue is because it cant reach coredns service/pods, which both are currently on 10.244.0.35 and 10.244.0.36 IPs
  
  ```
  NAME                                                                 READY   STATUS    RESTARTS         AGE     IP              NODE        NOMINATED NODE   READINESS GATES
  calico-kube-controllers-7b564c7d85-rkc98                             1/1     Running   0                278d    10.244.0.30     Master       <none>           <none>
  canal-5cffv                                                          2/2     Running   4 (378d ago)     630d    10.203.8.167    Worker3      <none>           <none>
  canal-p822m                                                          2/2     Running   2 (474d ago)     630d    10.203.8.165    Master       <none>           <none>
  canal-scdrs                                                          2/2     Running   4 (351d ago)     630d    10.203.40.166   Worker2      <none>           <none>
  canal-xkx7f                                                          2/2     Running   0                5h16m   10.203.8.174    Worker4      <none>           <none>
  coredns-cc4c6799f-4tjbd                                              1/1     Running   0                11h     10.244.0.36     Master       <none>           <none>
  coredns-cc4c6799f-bkq8n                                              1/1     Running   0                11h     10.244.0.35     Master       <none>           <none>
  etcd-Master                                                          1/1     Running   6 (474d ago)     630d    10.203.8.165    Master       <none>           <none>
  kube-apiserver-Master                                                1/1     Running   6 (474d ago)     630d    10.203.8.165    Master       <none>           <none>
  kube-controller-manager-Master                                       1/1     Running   10 (265d ago)    392d    10.203.8.165    Master       <none>           <none>
  kube-proxy-9n4wc                                                     1/1     Running   1 (474d ago)     630d    10.203.8.165    Master       <none>           <none>
  kube-proxy-rtccq                                                     1/1     Running   2 (351d ago)     630d    10.203.40.166   Worker2      <none>           <none>
  kube-proxy-vtr55                                                     1/1     Running   0                5h16m   10.203.8.174    Worker4      <none>           <none>
  kube-proxy-w9wkk                                                     1/1     Running   2 (378d ago)     630d    10.203.8.167    Worker3      <none>           <none>
  kube-scheduler-Master                                                1/1     Running   326 (265d ago)   630d    10.203.8.165    Master       <none>           <none>
  kube-state-metrics-8945855d-pvqqt                                    1/1     Running   0                139d    10.244.2.89     Worker3      <none>           <none>
  metrics-server-7cf8b65d65-qpnzm                                      1/1     Running   2 (6h43m ago)    139d    10.244.2.83     Worker3      <none>           <none>
  
  ```
  
  Running dnsutils on Worker2 for example
  
  ```
  root@dnsutils2:/# nslookup kubernetes.default
  Server:     10.96.0.10
  Address:    10.96.0.10#53
  
  Name:   kubernetes.default.svc.cluster.local
  Address: 10.96.0.1
  
  root@dnsutils2:/# ping 10.244.0.0
  PING 10.244.0.0 (10.244.0.0) 56(84) bytes of data.
  64 bytes from 10.244.0.0: icmp_seq=1 ttl=63 time=0.787 ms
  64 bytes from 10.244.0.0: icmp_seq=2 ttl=63 time=0.835 ms
  ^C
  --- 10.244.0.0 ping statistics ---
  2 packets transmitted, 2 received, 0% packet loss, time 1061ms
  rtt min/avg/max/mdev = 0.787/0.811/0.835/0.024 ms
  root@dnsutils2:/# ping 10.244.1.0
  PING 10.244.1.0 (10.244.1.0) 56(84) bytes of data.
  64 bytes from 10.244.1.0: icmp_seq=1 ttl=64 time=0.129 ms
  64 bytes from 10.244.1.0: icmp_seq=2 ttl=64 time=0.152 ms
  ^C
  --- 10.244.1.0 ping statistics ---
  2 packets transmitted, 2 received, 0% packet loss, time 1021ms
  rtt min/avg/max/mdev = 0.129/0.140/0.152/0.016 ms
  root@dnsutils2:/# ping 10.244.2.0
  PING 10.244.2.0 (10.244.2.0) 56(84) bytes of data.
  64 bytes from 10.244.2.0: icmp_seq=1 ttl=63 time=1.03 ms
  64 bytes from 10.244.2.0: icmp_seq=2 ttl=63 time=0.933 ms
  ^C
  --- 10.244.2.0 ping statistics ---
  2 packets transmitted, 2 received, 0% packet loss, time 1001ms
  rtt min/avg/max/mdev = 0.933/0.984/1.036/0.060 ms
  root@dnsutils2:/# ping 10.244.5.0
  PING 10.244.5.0 (10.244.5.0) 56(84) bytes of data.
  64 bytes from 10.244.5.0: icmp_seq=1 ttl=63 time=0.787 ms
  64 bytes from 10.244.5.0: icmp_seq=2 ttl=63 time=0.995 ms
  ^C
  --- 10.244.5.0 ping statistics ---
  2 packets transmitted, 2 received, 0% packet loss, time 1024ms
  rtt min/avg/max/mdev = 0.787/0.891/0.995/0.104 ms
  
  ```
  
  Cluster was initialized running below command
  
  ```
  kubeadm init --apiserver-advertise-address=10.203.8.165 --pod-network-cidr=10.244.0.0/16
  
  ```
  
  And for network cni plugin flannel-canal was used
  
  ```
  kubectl apply -f https://projectcalico.docs.tigera.io/manifests/canal.yaml
  
  ```
  
  I would really appreciate any help on where to start debugging this.
  Basically this Worker3 was working just fine before a restart and admins say that nothing was changed from network side or security groups etc. This is running on premises on VMware VMs.
  
  I have also tried `kubeadm reset` and then `iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X` , rebooting the node and then `kubeadm join` but still the issue persists. No idea why this specific Worker3 wont ping anything besides 10.244.1.0 and 10.244.5.0 (itself)
  
  I am confused about why wouldnt node be able to ping 10.244.2.0 network for example, but is able to ping 10.244.1.0 although there is iptables
  
  ```
  -A FORWARD -s 10.244.0.0/16 -j ACCEPT
  -A FORWARD -d 10.244.0.0/16 -j ACCEPT
  
  ```
  
  and also routes
  
  ```
  # route -n
  Kernel IP routing table
  Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
  0.0.0.0         10.203.8.254    0.0.0.0         UG    100    0        0 ens192
  10.203.8.0      0.0.0.0         255.255.255.0   U     100    0        0 ens192
  10.244.0.0      10.244.0.0      255.255.255.0   UG    0      0        0 flannel.1
  10.244.1.0      10.244.1.0      255.255.255.0   UG    0      0        0 flannel.1
  10.244.2.0      10.244.2.0      255.255.255.0   UG    0      0        0 flannel.1
  10.244.5.2      0.0.0.0         255.255.255.255 UH    0      0        0 cali952544557fa
  10.244.5.3      0.0.0.0         255.255.255.255 UH    0      0        0 calic15b507754a
  10.244.5.4      0.0.0.0         255.255.255.255 UH    0      0        0 calif987de568c4
  10.244.5.7      0.0.0.0         255.255.255.255 UH    0      0        0 cali969530ca43d
  10.244.5.8      0.0.0.0         255.255.255.255 UH    0      0        0 calib3c61c3cba9
  
  ```
  
  **Tags:** kubernetes
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2023-12-20. Score: 0 | Accepted answer
  
  So after deleting all canal PODs and letting them recreate, everything started to work again as it should.
  Seems like the recreation of those PODs somehow recreated routes and added the Worker node. Not sure why it didnt work right after a worker reboot, later when removing and re-joining this worker to the cluster, it got assigned a new CIDR, after canal POD recreation it started to work.
  
  
  ---
  
  ### Answer by username_1 on 2023-12-14. Score: 0
  
  Do you have vxlan in canal.yaml? Then change that to host-gw and see if it fixes the problem. flannel will communicate new routes to the kernel with netlink sockets then.
  
  ```
    net-conf.json: |
      {
        "Network": "10.244.0.0/16",
        "Backend": {
          "Type": "vxlan" # CHANGE TO host-gw
        }
      }
  
  ```
  
  More info here.
  
  [kubernetes access nodeport fails, master cannot reach workers](https://devops.stackexchange.com/questions/18555/kubernetes-access-nodeport-fails-master-cannot-reach-workers/18557#18557)
  
  and on the flannel github.
  
  <https://github.com/flannel-io/flannel>
  
  If it still doesn't work, try adding the specific network interface flannel must work on. If you dont it will claim the first interface it finds.
  
  ```
   - name: kube-flannel
            image: quay.io/coreos/flannel:v0.9.1
            command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]
  
  ```
  
  add --iface=iface\_you\_want to "command"
  
  
  ---
  
  