Answers: <p>Kubelet would usually bind to whichever interface holds your default gateway.
  I suspect your SDN pods are failing to start up due to this. At that stage, I would check for <code>netstat</code>, <code>ip routes</code> and iptables rules.</p>
  <p>You may want to remove the internal network thing. Deploying Kubernetes, it is recommended for your nodes to run in a LAN, using loadbalancers to expose your ingresses.</p>
  <p>Otherwise, you need your default gateway to point to your internal network. And then, you'll need some kind of router, NAT-ing traffic while sending it back to your bridged network.</p>
  
Answers: CC BY-SA 4.0
Answers: 2022-10-07T18:32:36.713Z
Answers: 2022-10-07T18:32:36.713Z
Answers: SYN
Body: <p>Context:
  I am trying to setup a kubernetes cluster on my PC using virtual box.
  Here's the setup -
  In this setup i am able to launch pods from the control plane, as well as able to send http requests to pods
  <a href="https://i.sstatic.net/nn5Xa.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/nn5Xa.png" alt="enter image description here" /></a></p>
  <p>here CP01: master/control plane, W01  - worker1, W02 - worker2 node.</p>
  <p>I had initialized control plane using  -</p>
  <pre><code>master] kubeadm init --apiserver-advertise-address 10.5.5.1 --pod-network-cidr 10.244.0.0/16 --cri-socket unix:///var/run/cri-dockerd.sock
  worker] kubeadm join 10.5.5.1:6443  --token jv5pxe.t07snw8ewrbejn6i   --cri-socket  unix:///var/run/cri-dockerd.sock      --discovery-token-ca-cert-hash sha256:10fc6e3fdc2085085f1ea1a75c9eb4316f13759b0d3773377db86baa30d8b972
  </code></pre>
  <p>I am able to create deployment as -</p>
  <h2>run.yaml</h2>
  <pre class="lang-yml prettyprint-override"><code>apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: nginx-deployment
    labels:
      app: nginx
  spec:
    replicas: 2
    selector:
      matchLabels:
        app: nginx
    template:
      metadata:
        labels:
          app: nginx
      spec:
        containers:
        - name: nginx
          image: nginx:1.14.2
          ports:
          - containerPort: 80
  </code></pre>
  <p>Here's the load balancer -</p>
  <h2>serv.yaml</h2>
  <pre class="lang-yml prettyprint-override"><code>apiVersion: v1
  kind: Service
  metadata:
    name: hello-world
  spec:
    type: LoadBalancer
    selector:
      app: nginx
    ports:
      - name: http
        protocol: TCP
        port: 80
        targetPort: 80
  </code></pre>
  <p>and from the cp01 node , i am able to hit both the pods via load balancer</p>
  <pre><code>[root@cp01 ~]# kubectl get services
  NAME          TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
  hello-world   LoadBalancer   10.96.81.128   &lt;pending&gt;     80:31785/TCP   3m56s
  [root@cp01 ~]# curl 10.96.81.128|grep Welcome
     &lt;title&gt;Welcome to nginx!&lt;/title&gt;
     &lt;h1&gt;Welcome to nginx!&lt;/h1&gt;
  </code></pre>
  <p>I have following 2 Queries -</p>
  <p>Q1: What settings to i need to carry out to make the flannel work without enp0s3 interface?
  I don't need external connectivity on w01 and w02. So if i disable the enp0s3 on w01 and w02, the flannel pods on worker starts failing. Here's is how i tried reproducing issue on w01 -</p>
  <pre><code>[root@cp01 ~]# kubectl get pods -A
  NAMESPACE      NAME                           READY   STATUS    RESTARTS        AGE
  kube-flannel   kube-flannel-ds-mp8zs          1/1     Running   8 (31m ago)     41m
  kube-flannel   kube-flannel-ds-p5kwj          1/1     Running   2 (12h ago)     3d1h
  kube-flannel   kube-flannel-ds-wqpwl          1/1     Running   0               24m
  kube-system    coredns-565d847f94-xddkq       1/1     Running   1 (12h ago)     15h
  kube-system    coredns-565d847f94-xl7pj       1/1     Running   1 (12h ago)     15h
  kube-system    etcd-cp01                      1/1     Running   2 (12h ago)     3d1h
  kube-system    kube-apiserver-cp01            1/1     Running   2 (12h ago)     3d1h
  kube-system    kube-controller-manager-cp01   1/1     Running   2 (12h ago)     3d1h
  kube-system    kube-proxy-9f4xm               1/1     Running   2 (12h ago)     3d1h
  kube-system    kube-proxy-dhhqc               1/1     Running   2 (12h ago)     3d1h
  kube-system    kube-proxy-w64gc               1/1     Running   1 (2d16h ago)   3d1h
  kube-system    kube-scheduler-cp01            1/1     Running   2 (12h ago)     3d1h
  [root@cp01 ~]# ssh w01 'nmcli con down enp0s3'
  [root@cp01 ~]# kubectl delete pod -n kube-flannel kube-flannel-ds-mp8zs
  pod &quot;kube-flannel-ds-mp8zs&quot; deleted
  [root@cp01 ~]# kubectl delete pod -n kube-flannel kube-flannel-ds-wqpwl
  pod &quot;kube-flannel-ds-wqpwl&quot; deleted
  [root@cp01 ~]# kubectl get pods -A
  NAMESPACE      NAME                           READY   STATUS             RESTARTS        AGE
  kube-flannel   kube-flannel-ds-2kqq5          0/1     CrashLoopBackOff   2 (25s ago)     45s
  kube-flannel   kube-flannel-ds-kcwk6          1/1     Running            0               49s
  kube-flannel   kube-flannel-ds-p5kwj          1/1     Running            2 (12h ago)     3d1h
  kube-system    coredns-565d847f94-xddkq       1/1     Running            1 (12h ago)     15h
  kube-system    coredns-565d847f94-xl7pj       1/1     Running            1 (12h ago)     15h
  kube-system    etcd-cp01                      1/1     Running            2 (12h ago)     3d1h
  kube-system    kube-apiserver-cp01            1/1     Running            2 (12h ago)     3d1h
  kube-system    kube-controller-manager-cp01   1/1     Running            2 (12h ago)     3d1h
  kube-system    kube-proxy-9f4xm               1/1     Running            2 (12h ago)     3d1h
  kube-system    kube-proxy-dhhqc               1/1     Running            2 (12h ago)     3d1h
  kube-system    kube-proxy-w64gc               1/1     Running            1 (2d16h ago)   3d1h
  kube-system    kube-scheduler-cp01            1/1     Running            2 (12h ago)     3d1h
  </code></pre>
  <p>here's the reason -</p>
  <pre><code>[root@cp01 ~]# kubectl logs -n kube-flannel kube-flannel-ds-2kqq5
  Defaulted container &quot;kube-flannel&quot; out of: kube-flannel, install-cni-plugin (init), install-cni (init)
  I1005 08:17:59.211331       1 main.go:207] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true ifaceCanReach: subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
  W1005 08:17:59.211537       1 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
  E1005 08:17:59.213916       1 main.go:224] Failed to create SubnetManager: error retrieving pod spec for 'kube-flannel/kube-flannel-ds-2kqq5': Get &quot;https://10.96.0.1:443/api/v1/namespaces/kube-flannel/pods/kube-flannel-ds-2kqq5&quot;: dial tcp 10.96.0.1:443: connect: network is unreachable
  </code></pre>
  <p>Q2: I plan to send http request to the load balancer via enp0s3 interface on cp01 node.
  do i need to :</p>
  <ul>
  <li>reset the cluster and do the kube init again using 0.0.0.0 ip or,</li>
  <li>is there a way to accomplish it without disturbing existing configurations/setup (using ingress?) ?</li>
  </ul>
  <p>Please advice.
  I have started learning kubernetes recently, so please excuse if i missed out on some basic concepts of the kubernetes world while framing these queries.
  Please do let me know if something is unclear.</p>
  <p>UPDATE:</p>
  <p>Q2:
  i tried initializing kubeadm node via -</p>
  <pre><code>[root@cp01 ~]# kubeadm init --apiserver-advertise-address 0.0.0.0  --pod-network-cidr 10.244.0.0/16 --cri-socket unix:///var/run/cri-dockerd.sock
  </code></pre>
  <p>but with this method, worker nodes are unable to join cluster from 10.5.5.0/24 (enp0s8) network. they are able to join cluster as -</p>
  <pre><code>kubeadm join 192.168.29.73:6443 --cri-socket  unix:///var/run/cri-dockerd.sock   --token 6srhn0.1lyiffiml08qcnfw --discovery-token-ca-cert-hash sha256:ab87e3e04da65176725776c08e0f924bbc07b26d0f8e2501793067e477ab6379
  </code></pre>
  
Tags: <kubernetes>
Title: forward http request to pod running on worker node
UserDisplayName: puneet336
ThreadText: # forward http request to pod running on worker node
  
  (Asked by: username_0 on 2022-10-05)
  
  Context:
  I am trying to setup a kubernetes cluster on my PC using virtual box.
  Here's the setup -
  In this setup i am able to launch pods from the control plane, as well as able to send http requests to pods
  [![enter image description here](https://i.sstatic.net/nn5Xa.png)](https://i.sstatic.net/nn5Xa.png)
  
  here CP01: master/control plane, W01 - worker1, W02 - worker2 node.
  
  I had initialized control plane using -
  
  ```
  master] kubeadm init --apiserver-advertise-address 10.5.5.1 --pod-network-cidr 10.244.0.0/16 --cri-socket unix:///var/run/cri-dockerd.sock
  worker] kubeadm join 10.5.5.1:6443  --token jv5pxe.t07snw8ewrbejn6i   --cri-socket  unix:///var/run/cri-dockerd.sock      --discovery-token-ca-cert-hash sha256:10fc6e3fdc2085085f1ea1a75c9eb4316f13759b0d3773377db86baa30d8b972
  
  ```
  
  I am able to create deployment as -
  
  ## run.yaml
  
  ```yml
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: nginx-deployment
    labels:
      app: nginx
  spec:
    replicas: 2
    selector:
      matchLabels:
        app: nginx
    template:
      metadata:
        labels:
          app: nginx
      spec:
        containers:
        - name: nginx
          image: nginx:1.14.2
          ports:
          - containerPort: 80
  
  ```
  
  Here's the load balancer -
  
  ## serv.yaml
  
  ```yml
  apiVersion: v1
  kind: Service
  metadata:
    name: hello-world
  spec:
    type: LoadBalancer
    selector:
      app: nginx
    ports:
      - name: http
        protocol: TCP
        port: 80
        targetPort: 80
  
  ```
  
  and from the cp01 node , i am able to hit both the pods via load balancer
  
  ```
  [root@cp01 ~]# kubectl get services
  NAME          TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
  hello-world   LoadBalancer   10.96.81.128   <pending>     80:31785/TCP   3m56s
  [root@cp01 ~]# curl 10.96.81.128|grep Welcome
     <title>Welcome to nginx!</title>
     <h1>Welcome to nginx!</h1>
  
  ```
  
  I have following 2 Queries -
  
  Q1: What settings to i need to carry out to make the flannel work without enp0s3 interface?
  I don't need external connectivity on w01 and w02. So if i disable the enp0s3 on w01 and w02, the flannel pods on worker starts failing. Here's is how i tried reproducing issue on w01 -
  
  ```
  [root@cp01 ~]# kubectl get pods -A
  NAMESPACE      NAME                           READY   STATUS    RESTARTS        AGE
  kube-flannel   kube-flannel-ds-mp8zs          1/1     Running   8 (31m ago)     41m
  kube-flannel   kube-flannel-ds-p5kwj          1/1     Running   2 (12h ago)     3d1h
  kube-flannel   kube-flannel-ds-wqpwl          1/1     Running   0               24m
  kube-system    coredns-565d847f94-xddkq       1/1     Running   1 (12h ago)     15h
  kube-system    coredns-565d847f94-xl7pj       1/1     Running   1 (12h ago)     15h
  kube-system    etcd-cp01                      1/1     Running   2 (12h ago)     3d1h
  kube-system    kube-apiserver-cp01            1/1     Running   2 (12h ago)     3d1h
  kube-system    kube-controller-manager-cp01   1/1     Running   2 (12h ago)     3d1h
  kube-system    kube-proxy-9f4xm               1/1     Running   2 (12h ago)     3d1h
  kube-system    kube-proxy-dhhqc               1/1     Running   2 (12h ago)     3d1h
  kube-system    kube-proxy-w64gc               1/1     Running   1 (2d16h ago)   3d1h
  kube-system    kube-scheduler-cp01            1/1     Running   2 (12h ago)     3d1h
  [root@cp01 ~]# ssh w01 'nmcli con down enp0s3'
  [root@cp01 ~]# kubectl delete pod -n kube-flannel kube-flannel-ds-mp8zs
  pod "kube-flannel-ds-mp8zs" deleted
  [root@cp01 ~]# kubectl delete pod -n kube-flannel kube-flannel-ds-wqpwl
  pod "kube-flannel-ds-wqpwl" deleted
  [root@cp01 ~]# kubectl get pods -A
  NAMESPACE      NAME                           READY   STATUS             RESTARTS        AGE
  kube-flannel   kube-flannel-ds-2kqq5          0/1     CrashLoopBackOff   2 (25s ago)     45s
  kube-flannel   kube-flannel-ds-kcwk6          1/1     Running            0               49s
  kube-flannel   kube-flannel-ds-p5kwj          1/1     Running            2 (12h ago)     3d1h
  kube-system    coredns-565d847f94-xddkq       1/1     Running            1 (12h ago)     15h
  kube-system    coredns-565d847f94-xl7pj       1/1     Running            1 (12h ago)     15h
  kube-system    etcd-cp01                      1/1     Running            2 (12h ago)     3d1h
  kube-system    kube-apiserver-cp01            1/1     Running            2 (12h ago)     3d1h
  kube-system    kube-controller-manager-cp01   1/1     Running            2 (12h ago)     3d1h
  kube-system    kube-proxy-9f4xm               1/1     Running            2 (12h ago)     3d1h
  kube-system    kube-proxy-dhhqc               1/1     Running            2 (12h ago)     3d1h
  kube-system    kube-proxy-w64gc               1/1     Running            1 (2d16h ago)   3d1h
  kube-system    kube-scheduler-cp01            1/1     Running            2 (12h ago)     3d1h
  
  ```
  
  here's the reason -
  
  ```
  [root@cp01 ~]# kubectl logs -n kube-flannel kube-flannel-ds-2kqq5
  Defaulted container "kube-flannel" out of: kube-flannel, install-cni-plugin (init), install-cni (init)
  I1005 08:17:59.211331       1 main.go:207] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true ifaceCanReach: subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
  W1005 08:17:59.211537       1 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
  E1005 08:17:59.213916       1 main.go:224] Failed to create SubnetManager: error retrieving pod spec for 'kube-flannel/kube-flannel-ds-2kqq5': Get "https://10.96.0.1:443/api/v1/namespaces/kube-flannel/pods/kube-flannel-ds-2kqq5": dial tcp 10.96.0.1:443: connect: network is unreachable
  
  ```
  
  Q2: I plan to send http request to the load balancer via enp0s3 interface on cp01 node.
  do i need to :
  
  * reset the cluster and do the kube init again using 0.0.0.0 ip or,
  * is there a way to accomplish it without disturbing existing configurations/setup (using ingress?) ?
  
  Please advice.
  I have started learning kubernetes recently, so please excuse if i missed out on some basic concepts of the kubernetes world while framing these queries.
  Please do let me know if something is unclear.
  
  UPDATE:
  
  Q2:
  i tried initializing kubeadm node via -
  
  ```
  [root@cp01 ~]# kubeadm init --apiserver-advertise-address 0.0.0.0  --pod-network-cidr 10.244.0.0/16 --cri-socket unix:///var/run/cri-dockerd.sock
  
  ```
  
  but with this method, worker nodes are unable to join cluster from 10.5.5.0/24 (enp0s8) network. they are able to join cluster as -
  
  ```
  kubeadm join 192.168.29.73:6443 --cri-socket  unix:///var/run/cri-dockerd.sock   --token 6srhn0.1lyiffiml08qcnfw --discovery-token-ca-cert-hash sha256:ab87e3e04da65176725776c08e0f924bbc07b26d0f8e2501793067e477ab6379
  
  ```
  
  **Tags:** kubernetes
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-07. Score: 1
  
  Kubelet would usually bind to whichever interface holds your default gateway.
  I suspect your SDN pods are failing to start up due to this. At that stage, I would check for `netstat`, `ip routes` and iptables rules.
  
  You may want to remove the internal network thing. Deploying Kubernetes, it is recommended for your nodes to run in a LAN, using loadbalancers to expose your ingresses.
  
  Otherwise, you need your default gateway to point to your internal network. And then, you'll need some kind of router, NAT-ing traffic while sending it back to your bridged network.
  
  
  ---
  
  