Answers: <p>Found the cause, maybe someone will find it useful in the future.</p>
  <p>This happened because flannel is not correctly set up for an on premise kubernetes environment.</p>
  <p>I installed the default kube-flannel.yaml but two things need to be changed first in on premise environments.</p>
  <ol>
  <li><p>The specific ip of the network interface must be passed to flanneld as a commandline argument. Otherwise flannel takes the first network interface.</p>
  <pre><code> command:
   - /opt/bin/flanneld
   args:
   - --ip-masq
   - --kube-subnet-mgr
   - --iface=enp0s8
  </code></pre>
  </li>
  <li><p>The default config has &quot;backend type&quot; VXLAN which causes the vxlan.go package code to be triggered in flannel. This works in azure, gcloud and AWS etc. but not on premise. Type host-gw must be configured so the hostgw code is run which tries to connect to other machines with layer2 routing.</p>
  <p>net-conf.json: |
  {
  &quot;Network&quot;: &quot;10.244.0.0/16&quot;,
  &quot;Backend&quot;: {
  &quot;Type&quot;: &quot;host-gw&quot;
  }
  }</p>
  </li>
  </ol>
  <p>Checking out flannel's source code for hostgw:</p>
  <p><a href="https://github.com/flannel-io/flannel/blob/master/pkg/backend/hostgw/hostgw.go" rel="nofollow noreferrer">https://github.com/flannel-io/flannel/blob/master/pkg/backend/hostgw/hostgw.go</a></p>
  <p>every backend is added with it's &quot;constructor&quot; New function during startup so in the case of host-gw where ip address of extIface is the one passed with --iface option.</p>
  <pre><code>func init() {
      backend.Register(&quot;host-gw&quot;, New)
  }
     
  func New(sm subnet.Manager, extIface *backend.ExternalInterface) (backend.Backend, error) {
      if !extIface.ExtAddr.Equal(extIface.IfaceAddr) {
          return nil, fmt.Errorf(&quot;PublicIP differs from interface IP probably on NAT is not supported by host-gw&quot;)
      }
  
      be := &amp;HostgwBackend{
          sm:       sm,
          extIface: extIface,
      }
      return be, nil
  }
  </code></pre>
  <p>The HostgwBackend is then instantiated in main.go and RegisterNetwork is called.</p>
  <pre><code>bm := backend.NewManager(ctx, sm, extIface)
  be, err := bm.GetBackend(config.BackendType)
  bn, err := be.RegisterNetwork(ctx, &amp;wg, config)
  </code></pre>
  <p>And in RegisterNetwork the function is set up which communicates the pod routes to the &quot;pod-network-cidr&quot; (10.244.0.0/24) of the cluster to the kernel with the netlink package when it calls AcquireLease and Lease. Then to retrieve the routes it uses netlink again to query for the routes.</p>
  <pre><code>    n.GetRoute = func(lease *lease.Lease) *netlink.Route {
          return &amp;netlink.Route{
              Dst:       lease.Subnet.ToIPNet(),
              Gw:        lease.Attrs.PublicIP.ToIP(),
              LinkIndex: n.LinkIndex,
          }
      }
  </code></pre>
  <p>So after initialization 'ip route' shows all the created routes.</p>
  <p>ip route show</p>
  <pre><code>10.244.1.0/24 via 10.203.32.120 dev ens160 
  10.244.2.0/24 via 10.203.32.98 dev ens160 
  10.244.3.0/24 via 10.203.32.111 dev ens160 
  10.244.4.0/24 via 10.203.32.99 dev ens160 
  10.244.5.0/24 via 10.203.32.110 dev ens160 
  10.244.6.0/24 via 10.203.32.121 dev ens160 
  </code></pre>
  
Answers: CC BY-SA 4.0
Answers: 2023-11-15T10:47:47.257Z
Answers: 2023-11-16T09:55:38.723Z
Answers: 2023-11-16T09:55:38.723Z
Answers: Serve Laurijssen
Body: <p>I've got an on premise kubernetes cluster with a NodePort service running on port 32001. But in 80% of the cases requests fail:</p>
  <pre><code>curl localhost:32001
  </code></pre>
  <p>times out and sometimes it works, results are returned</p>
  <p>Als starting an nginx pod (ng) in the cluster and calling</p>
  <pre><code>wget -O- http://ng:80
  </code></pre>
  <p>from a busybox does not work ever.
  While in another cluster it works.</p>
  <p>We use flannel CNI so I checked logs of flannel pods and kube-proxy but can't find anything weird.</p>
  <p>Any ideas what to check for? What can cause a sometimes timeout and sometimes success?</p>
  
Tags: <kubernetes><networking><containerd>
Title: kubernetes access nodeport fails, master cannot reach workers
UserDisplayName: Serve Laurijssen
ThreadText: # kubernetes access nodeport fails, master cannot reach workers
  
  (Asked by: username_0 on 2023-11-15)
  
  I've got an on premise kubernetes cluster with a NodePort service running on port 32001. But in 80% of the cases requests fail:
  
  ```
  curl localhost:32001
  
  ```
  
  times out and sometimes it works, results are returned
  
  Als starting an nginx pod (ng) in the cluster and calling
  
  ```
  wget -O- http://ng:80
  
  ```
  
  from a busybox does not work ever.
  While in another cluster it works.
  
  We use flannel CNI so I checked logs of flannel pods and kube-proxy but can't find anything weird.
  
  Any ideas what to check for? What can cause a sometimes timeout and sometimes success?
  
  **Tags:** kubernetes, networking, containerd
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-11-15. Score: 1 | Accepted answer
  
  Found the cause, maybe someone will find it useful in the future.
  
  This happened because flannel is not correctly set up for an on premise kubernetes environment.
  
  I installed the default kube-flannel.yaml but two things need to be changed first in on premise environments.
  
  1. The specific ip of the network interface must be passed to flanneld as a commandline argument. Otherwise flannel takes the first network interface.
  
     ```
      command:
      - /opt/bin/flanneld
      args:
      - --ip-masq
      - --kube-subnet-mgr
      - --iface=enp0s8
  
     ```
  2. The default config has "backend type" VXLAN which causes the vxlan.go package code to be triggered in flannel. This works in azure, gcloud and AWS etc. but not on premise. Type host-gw must be configured so the hostgw code is run which tries to connect to other machines with layer2 routing.
  
     net-conf.json: |
     {
     "Network": "10.244.0.0/16",
     "Backend": {
     "Type": "host-gw"
     }
     }
  
  Checking out flannel's source code for hostgw:
  
  <https://github.com/flannel-io/flannel/blob/master/pkg/backend/hostgw/hostgw.go>
  
  every backend is added with it's "constructor" New function during startup so in the case of host-gw where ip address of extIface is the one passed with --iface option.
  
  ```
  func init() {
      backend.Register("host-gw", New)
  }
     
  func New(sm subnet.Manager, extIface *backend.ExternalInterface) (backend.Backend, error) {
      if !extIface.ExtAddr.Equal(extIface.IfaceAddr) {
          return nil, fmt.Errorf("PublicIP differs from interface IP probably on NAT is not supported by host-gw")
      }
  
      be := &HostgwBackend{
          sm:       sm,
          extIface: extIface,
      }
      return be, nil
  }
  
  ```
  
  The HostgwBackend is then instantiated in main.go and RegisterNetwork is called.
  
  ```
  bm := backend.NewManager(ctx, sm, extIface)
  be, err := bm.GetBackend(config.BackendType)
  bn, err := be.RegisterNetwork(ctx, &wg, config)
  
  ```
  
  And in RegisterNetwork the function is set up which communicates the pod routes to the "pod-network-cidr" (10.244.0.0/24) of the cluster to the kernel with the netlink package when it calls AcquireLease and Lease. Then to retrieve the routes it uses netlink again to query for the routes.
  
  ```
      n.GetRoute = func(lease *lease.Lease) *netlink.Route {
          return &netlink.Route{
              Dst:       lease.Subnet.ToIPNet(),
              Gw:        lease.Attrs.PublicIP.ToIP(),
              LinkIndex: n.LinkIndex,
          }
      }
  
  ```
  
  So after initialization 'ip route' shows all the created routes.
  
  ip route show
  
  ```
  10.244.1.0/24 via 10.203.32.120 dev ens160 
  10.244.2.0/24 via 10.203.32.98 dev ens160 
  10.244.3.0/24 via 10.203.32.111 dev ens160 
  10.244.4.0/24 via 10.203.32.99 dev ens160 
  10.244.5.0/24 via 10.203.32.110 dev ens160 
  10.244.6.0/24 via 10.203.32.121 dev ens160 
  
  ```
  
  
  ---
  
  