Body: <p>I have a working HA distributed hybrid Kubernetes (K3S) cluster with three masters running on VPS nodes.
  They have public IPs and no private network (nodes spread across regions and cloud providers).</p>
  <p>Now I want to add several agent nodes (workers) located on-pemises without public IP address.</p>
  <p>Existing cluster uses embedded ETCD and a native wireguard to establish a VPN mesh for cluster traffic via public IPs.
  Nodes have been started with following parameters (<a href="https://docs.k3s.io/networking/distributed-multicloud" rel="nofollow noreferrer">see documentation</a>):</p>
  <pre><code>--node-external-ip=&lt;SERVER_EXTERNAL_IP&gt; --flannel-backend=wireguard-native --flannel-external-ip
  </code></pre>
  <p>That means, they use default interface (e.g. eth0) and public IPs for VPN mesh network.</p>
  <p>Now, I'd like to use Tailscale network to add K3s agents to that running setup.<br />
  Tailscale is installed on all master and agent nodes and all joined a Tailscale's network.<br />
  Tailscale's network interface is called <code>tailscale0</code>.<br />
  Native K3S Integration with the Tailscale VPN provider can't be used due to embedded ETCD used in the cluster.</p>
  <p>Level 1 question: how do I add agents to above mentioned existing setup via tailscale network?
  Level 2 question: is it possible to do that without interrupting control-plane that uses embedded ETCD?</p>
  <hr />
  <p>So far, I found following example in Internet (not sure it is relevant to K3s):</p>
  <ul>
  <li>configure master(s) with following parameters:</li>
  </ul>
  <pre><code>--node-external-ip &lt;PUBLIC_IP&gt; --advertise-address &lt;TAILSCALE_IP_NODE_1&gt; --flannel-iface tailscale0
  </code></pre>
  <ul>
  <li>install K3S agent node with</li>
  </ul>
  <pre><code>K3S_URL=https://&lt;TAILSCALE_IP_NODE_1&gt;:6443 \ 
  K3S_TOKEN=&quot;&lt;replaced&gt;&quot; \
  sh -s - \
  --node-ip &lt;TAILSCALE_IP_NODE_2&gt; \
  --flannel-iface tailscale0
  </code></pre>
  <p>I've not tested it yet and I don't know how to apply this correctly on an existing HA cluster without interrupting running ETCD.
  Just don't want to kill a running cluster.</p>
  
Comments: CC BY-SA 4.0
Comments: 2024-05-06T09:09:09.987Z
Comments: I'm facing a similar issue, at this time I can only suggest to try it on a replicated scenario, because the embedded etcd is kinda tricky. Few days ago I've tried to add a second master node and the whole cluster failed because the second master node couldn't reach the master etcd node behind tailscale (at least without any extra configuration). Try to replicate the situation first, I suggest to use lima (linux machines https://github.com/lima-vm/lima) or multipass https://multipass.run/
Comments: ngeorger
Comments: CC BY-SA 4.0
Comments: 2024-05-07T07:44:24.350Z
Comments: thnks for your input. I've experimented on test setup. To my opinion, there is no way to do it correctly. It is possible, I've added workers behind NAT, but when I use tailscale to join the cluster, it introduces MTU mismatch and the workload on the agent works not stable. At least when there is high data exchange like using distributed filesystem. Maybe it'll work if all agents will run on tailscale and no workload on master nodes.
Comments: Ostap
Tags: <kubernetes><k3s>
Title: Kubernetes (K3S) Masters on cloud and workers behind NAT
UserDisplayName: Ostap
ThreadText: # Kubernetes (K3S) Masters on cloud and workers behind NAT
  
  (Asked by: username_0 on 2024-04-12)
  
  I have a working HA distributed hybrid Kubernetes (K3S) cluster with three masters running on VPS nodes.
  They have public IPs and no private network (nodes spread across regions and cloud providers).
  
  Now I want to add several agent nodes (workers) located on-pemises without public IP address.
  
  Existing cluster uses embedded ETCD and a native wireguard to establish a VPN mesh for cluster traffic via public IPs.
  Nodes have been started with following parameters ([see documentation](https://docs.k3s.io/networking/distributed-multicloud)):
  
  ```
  --node-external-ip=<SERVER_EXTERNAL_IP> --flannel-backend=wireguard-native --flannel-external-ip
  
  ```
  
  That means, they use default interface (e.g. eth0) and public IPs for VPN mesh network.
  
  Now, I'd like to use Tailscale network to add K3s agents to that running setup.  
  Tailscale is installed on all master and agent nodes and all joined a Tailscale's network.  
  Tailscale's network interface is called `tailscale0`.  
  Native K3S Integration with the Tailscale VPN provider can't be used due to embedded ETCD used in the cluster.
  
  Level 1 question: how do I add agents to above mentioned existing setup via tailscale network?
  Level 2 question: is it possible to do that without interrupting control-plane that uses embedded ETCD?
  
  ---
  
  So far, I found following example in Internet (not sure it is relevant to K3s):
  
  * configure master(s) with following parameters:
  
  ```
  --node-external-ip <PUBLIC_IP> --advertise-address <TAILSCALE_IP_NODE_1> --flannel-iface tailscale0
  
  ```
  
  * install K3S agent node with
  
  ```
  K3S_URL=https://<TAILSCALE_IP_NODE_1>:6443 \ 
  K3S_TOKEN="<replaced>" \
  sh -s - \
  --node-ip <TAILSCALE_IP_NODE_2> \
  --flannel-iface tailscale0
  
  ```
  
  I've not tested it yet and I don't know how to apply this correctly on an existing HA cluster without interrupting running ETCD.
  Just don't want to kill a running cluster.
  
  **Tags:** kubernetes, k3s
  
  > **Comments:**
  > **username_1**: I'm facing a similar issue, at this time I can only suggest to try it on a replicated scenario, because the embedded etcd is kinda tricky. Few days ago I've tried to add a second master node and the whole cluster failed because the second master node couldn't reach the master etcd node behind tailscale (at least without any extra configuration). Try to replicate the situation first, I suggest to use lima (linux machines https://github.com/lima-vm/lima) or multipass https://multipass.run/  
  > **username_0**: thnks for your input. I've experimented on test setup. To my opinion, there is no way to do it correctly. It is possible, I've added workers behind NAT, but when I use tailscale to join the cluster, it introduces MTU mismatch and the workload on the agent works not stable. At least when there is high data exchange like using distributed filesystem. Maybe it'll work if all agents will run on tailscale and no workload on master nodes.  
  
  ---
  
  ## 0 answers
  
  