Body: <p>I had set up a Kubernetes cluster on four Raspberry Pis for toy projects awhile back, and I originally set things up using Docker. But ever since Kubernetes decided to drop support for Docker I decided to re-do the cluster using ContainerD instead. On top of that, I wiped the SD cards and installed the x64 version of Raspberry Pi OS (instead of the 32-bit version which I was using previously). I've slowly been piecing together a runbook of how to install things but have encountered a snag: five minutes after calling <code>kubeadm init</code> on the master node, the cluster stops working. For reference, the internal IP of my master node is 192.168.1.194. Here's the full command I use to initialize things:</p>
  <pre><code>sudo kubeadm init --pod-network-cidr=10.244.0.0/16 \
      --token-ttl=0 --apiserver-advertise-address=192.168.1.194
  </code></pre>
  <p>After running that command, and subsequently copying the /etc/kubernetes/admin.conf file to ~/.kube/config, I can run the following command:</p>
  <pre><code>$ kubectl get nodes
  
  NAME           STATUS     ROLES                  AGE     VERSION
  k8s-master-1   NotReady   control-plane,master   3m36s   v1.23.4
  </code></pre>
  <p>And it will continue to show a <code>NotReady</code> status for about 5 minutes, after which point the same command yields a very different result:</p>
  <pre><code>$ kubectl get nodes
  
  The connection to the server 192.168.1.194:6443 was refused - did you specify the right host or port?
  </code></pre>
  <p>I'm not sure why this is happening, but it is very consistent. I have tried a few times now to <code>kubeadm reset</code> and then <code>kubeadm init</code> again, and the connection timeout always happens at the 5-minute mark. So the last time I tried to do this, I decided to tail all the log files under /var/log/containers/. After the 5-minute mark, it is repeatedly logging some variation of a connection error to 127.0.0.1:2379. For example:</p>
  <blockquote>
  <p>2022-03-09T19:30:29.307156643-06:00 stderr F W0310 01:30:29.306871       1 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1  0 }. Err: connection error: desc = &quot;transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused&quot;. Reconnecting...</p>
  </blockquote>
  <p>But prior to the 5-minute mark, it looks like a bunch of services start shutting down right at the end, followed by what looks like maybe etcd shuts down as well. I've uploaded the full logs from the time <code>kubeadm init</code> runs, up until the dreaded 5-minute mark, <a href="https://gist.github.com/soapergem/fd71b313473aa366e5bb44acdaccaab4" rel="nofollow noreferrer">as a Gist</a>.</p>
  <p>Why won't Kubernetes start on my Pi? What am I missing?</p>
  
Comments: CC BY-SA 4.0
Comments: 2022-03-16T16:28:32.560Z
Comments: does all the ports are open? could you do test all ports like  `telnet localhost 2379`.
  Ref docs for all the ports: https://kubernetes.io/docs/reference/ports-and-protocols/
Comments: Vinod
Comments: CC BY-SA 4.0
Comments: 2022-03-17T02:41:14.080Z
Comments: Yes, all ports are open. The telnet command succeeds during that first 5 minute window of initialization, while etcd is active.
Comments: soapergem
Comments: CC BY-SA 4.0
Comments: 2022-03-17T07:02:04.703Z
Comments: if all steps are correct and it is failing then what i can think of is `kubeadm join` is not able to bring this node in cluster. there could be multiple reason for that. may be you could try to specify the `--control-plane-endpoint` while configuring the cluster which is basically your cluster endpoint. 
  Ref: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#:~:text=While%20%2D%2Dapiserver%2Dadvertise,supported%20by%20kubeadm
Comments: Vinod
Comments: CC BY-SA 4.0
Comments: 2022-03-21T02:52:44.587Z
Comments: Again, this is not one of the nodes but the master, so `kubeadm join` is not used.
Comments: soapergem
Tags: <kubernetes><containerd><raspberry-pi-os><etcd>
Title: Kubernetes cluster on Raspberry Pi stops working 5 minutes after init
UserDisplayName: soapergem
ThreadText: # Kubernetes cluster on Raspberry Pi stops working 5 minutes after init
  
  (Asked by: username_0 on 2022-03-10)
  
  I had set up a Kubernetes cluster on four Raspberry Pis for toy projects awhile back, and I originally set things up using Docker. But ever since Kubernetes decided to drop support for Docker I decided to re-do the cluster using ContainerD instead. On top of that, I wiped the SD cards and installed the x64 version of Raspberry Pi OS (instead of the 32-bit version which I was using previously). I've slowly been piecing together a runbook of how to install things but have encountered a snag: five minutes after calling `kubeadm init` on the master node, the cluster stops working. For reference, the internal IP of my master node is 192.168.1.194. Here's the full command I use to initialize things:
  
  ```
  sudo kubeadm init --pod-network-cidr=10.244.0.0/16 \
      --token-ttl=0 --apiserver-advertise-address=192.168.1.194
  
  ```
  
  After running that command, and subsequently copying the /etc/kubernetes/admin.conf file to ~/.kube/config, I can run the following command:
  
  ```
  $ kubectl get nodes
  
  NAME           STATUS     ROLES                  AGE     VERSION
  k8s-master-1   NotReady   control-plane,master   3m36s   v1.23.4
  
  ```
  
  And it will continue to show a `NotReady` status for about 5 minutes, after which point the same command yields a very different result:
  
  ```
  $ kubectl get nodes
  
  The connection to the server 192.168.1.194:6443 was refused - did you specify the right host or port?
  
  ```
  
  I'm not sure why this is happening, but it is very consistent. I have tried a few times now to `kubeadm reset` and then `kubeadm init` again, and the connection timeout always happens at the 5-minute mark. So the last time I tried to do this, I decided to tail all the log files under /var/log/containers/. After the 5-minute mark, it is repeatedly logging some variation of a connection error to 127.0.0.1:2379. For example:
  
  > 2022-03-09T19:30:29.307156643-06:00 stderr F W0310 01:30:29.306871 1 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 0 }. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
  
  But prior to the 5-minute mark, it looks like a bunch of services start shutting down right at the end, followed by what looks like maybe etcd shuts down as well. I've uploaded the full logs from the time `kubeadm init` runs, up until the dreaded 5-minute mark, [as a Gist](https://gist.github.com/username_0/fd71b313473aa366e5bb44acdaccaab4).
  
  Why won't Kubernetes start on my Pi? What am I missing?
  
  **Tags:** kubernetes, containerd, raspberry-pi-os, etcd
  
  > **Comments:**
  > **username_1**: does all the ports are open? could you do test all ports like  `telnet localhost 2379`. Ref docs for all the ports: https://kubernetes.io/docs/reference/ports-and-protocols/  
  > **username_0**: Yes, all ports are open. The telnet command succeeds during that first 5 minute window of initialization, while etcd is active.  
  > **username_1**: if all steps are correct and it is failing then what i can think of is `kubeadm join` is not able to bring this node in cluster. there could be multiple reason for that. may be you could try to specify the `--control-plane-endpoint` while configuring the cluster which is basically your cluster endpoint.  Ref: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#:~:text=While%20%2D%2Dapiserver%2Dadvertise,supported%20by%20kubeadm  
  > **username_0**: Again, this is not one of the nodes but the master, so `kubeadm join` is not used.  
  
  ---
  
  ## 0 answers
  
  