Answers: <p>Basically you did everything correctly.
  The chosen way to make your service reachable from the outside does not allow you to use specific, low, ports in order to keep isolation between services.
  You have two options:</p>
  <ol>
  <li>use a service of type LoadBalancer which will allocate a separate IP for the service, providing isolation between services and allowing you a specific port like 80 or 443</li>
  <li>modify the ingress deployment and use a <code>hostPort</code> for the nginx containers, you can then use the node IPs where a pod is running. This can lead to a lot of headaches (f.e. when updating the deployment) but might be an easy solution to get started</li>
  </ol>
  
Answers: CC BY-SA 4.0
Answers: 2024-01-09T07:48:20.557Z
Answers: Thanks for you answer, I fixed my problem using MetalLB with a super basic config.
  I misunderstood the ingress resource behaviour because of his IP address… I still don’t understand why the ingress resource has an IP assigned since its only purpose is to provide an nginx configuration to the ingress-nginx-controller service.
  Everything is now working as expected, thanks anyway for your help!
Answers: Lorenzo Rottigni
Answers: CC BY-SA 4.0
Answers: 2024-01-09T19:58:30.833Z
Answers: If you are using MetalLB and It is configured correctly it should provide you with a dedicated IP as it is a LoadBalancer implementation. Please accept the answer if it solved your issue.
Answers: Thomas
Answers: CC BY-SA 4.0
Answers: 2024-01-07T18:07:16.430Z
Answers: 2024-01-07T18:07:16.430Z
Answers: Thomas
Body: <p>I'm attempting to set up my own bare-metal Kubernetes cluster composed of a control plane (VPS 149.62.186.95) and a worker node (VPS 149.62.189.139). Both my VPS are running Ubuntu 22.04 with 2 vCPUs and 4GB vRAM.</p>
  <p>After installing Kubernetes (<code>kubeadm</code>) on both, I ran the following commands on my control plane:</p>
  <pre><code>kubeadm init
  
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
  
  # Weave deamonset
  kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml
  
  # baremetal nginx ingress
  kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml
  </code></pre>
  <p>I edited my DNS records as follows:</p>
  <ul>
  <li>registry.rottigni.tech -&gt; 149.62.186.95 (control plane)</li>
  <li>sandbox.rottigni.tech -&gt; 149.62.186.95 (control plane)</li>
  </ul>
  <p>I generated SSL certs for these domains on my control plane and stored them in Kubernetes secrets.</p>
  <p>Then, I applied my changes using kubectl apply -R -f . inside my repository. Here's the most relevant file, ingress.yml:</p>
  <pre><code>apiVersion: networking.k8s.io/v1
  kind: IngressClass
  metadata:
    name: nginx-ingress-class
  spec:
    controller: k8s.io/ingress-nginx
  ---
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: lr-labs-ingress
    namespace: lr-labs
    annotations:
      # cert-manager.io/issuer: &quot;letsencrypt-prod&quot;
      nginx.ingress.kubernetes.io/rewrite-target: /$1
  spec:
    ingressClassName: nginx-ingress-class
    tls:
      - hosts:
          - registry.rottigni.tech
        secretName: registry-ssl
      - hosts:
          - sandbox.rottigni.tech
        secretName: sandbox-ssl
  
    rules:
      - host: registry.rottigni.tech
        http:
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: registry-rottigni
                  port:
                    number: 5000
  
      - host: sandbox.rottigni.tech
        http:
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: sandbox-rottigni
                  port:
                    number: 80
  </code></pre>
  <p>At this point, everything seems to be working on the kubectl side, according to the following output:</p>
  <pre><code>Every 2.0s: kubectl get pods,deploy,svc,ing,endpoints,node -A                                                                                                                                                                           
  
  NAMESPACE       NAME                                                READY   STATUS      RESTARTS        AGE
  ingress-nginx   pod/ingress-nginx-admission-create-wn7xr            0/1     Completed   0               7h58m
  ingress-nginx   pod/ingress-nginx-admission-patch-dtzxf             0/1     Completed   0               7h58m
  ingress-nginx   pod/ingress-nginx-controller-86968d59df-vdv9f       1/1     Running     0               6h52m
  kube-system     pod/coredns-5dd5756b68-h4jwb                        1/1     Running     0               7h58m
  kube-system     pod/coredns-5dd5756b68-jrsdq                        1/1     Running     0               7h58m
  kube-system     pod/etcd-epsilon-rottigni-tech                      1/1     Running     0               7h58m
  kube-system     pod/kube-apiserver-epsilon-rottigni-tech            1/1     Running     0               7h58m
  kube-system     pod/kube-controller-manager-epsilon-rottigni-tech   1/1     Running     0               7h58m
  kube-system     pod/kube-proxy-4v8bc                                1/1     Running     0               7h58m
  kube-system     pod/kube-proxy-8hpjj                                1/1     Running     0               7h57m
  kube-system     pod/kube-scheduler-epsilon-rottigni-tech            1/1     Running     0               7h58m
  kube-system     pod/weave-net-gvzk5                                 2/2     Running     1 (7h58m ago)   7h58m
  kube-system     pod/weave-net-tbr92                                 2/2     Running     0               7h57m
  lr-labs         pod/registry-rottigni-5c6cfd877f-qqxsw              1/1     Running     0               7h54m
  lr-labs         pod/sandbox-rottigni-9668f7447-dv7js                1/1     Running     0               7h54m
  
  NAMESPACE       NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
  ingress-nginx   deployment.apps/ingress-nginx-controller   1/1     1            1           7h58m
  kube-system     deployment.apps/coredns                    2/2     2            2           7h58m
  lr-labs         deployment.apps/registry-rottigni          1/1     1            1           7h54m
  lr-labs         deployment.apps/sandbox-rottigni           1/1     1            1           7h54m
  
  NAMESPACE       NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
  default         service/kubernetes                           ClusterIP   10.96.0.1        &lt;none&gt;        443/TCP                      7h58m
  ingress-nginx   service/ingress-nginx-controller             NodePort    10.108.71.135    &lt;none&gt;        80:32095/TCP,443:30221/TCP   7h58m
  ingress-nginx   service/ingress-nginx-controller-admission   ClusterIP   10.107.203.218   &lt;none&gt;        443/TCP                      7h58m
  kube-system     service/kube-dns                             ClusterIP   10.96.0.10       &lt;none&gt;        53/UDP,53/TCP,9153/TCP       7h58m
  lr-labs         service/registry-rottigni                    NodePort    10.106.231.17    &lt;none&gt;        5000:31673/TCP               7h54m
  lr-labs         service/sandbox-rottigni                     NodePort    10.111.173.239   &lt;none&gt;        80:31607/TCP                 7h54m
  
  NAMESPACE   NAME                                        CLASS                 HOSTS                                          ADDRESS          PORTS     AGE
  lr-labs     ingress.networking.k8s.io/lr-labs-ingress   nginx-ingress-class   registry.rottigni.tech,sandbox.rottigni.tech   149.62.189.139   80, 443   7h54m
  
  NAMESPACE       NAME                                           ENDPOINTS                                            AGE
  default         endpoints/kubernetes                           149.62.186.95:6443                                   7h58m
  ingress-nginx   endpoints/ingress-nginx-controller             10.44.0.2:443,10.44.0.2:80                           7h58m
  ingress-nginx   endpoints/ingress-nginx-controller-admission   10.44.0.2:8443                                       7h58m
  kube-system     endpoints/kube-dns                             10.32.0.2:53,10.32.0.3:53,10.32.0.2:53 + 3 more...   7h58m
  lr-labs         endpoints/registry-rottigni                    10.44.0.3:5000                                       7h54m
  lr-labs         endpoints/sandbox-rottigni                     10.44.0.1:80                                         7h54m
  
  NAMESPACE   NAME                         STATUS   ROLES           AGE     VERSION
              node/epsilon-rottigni-tech   Ready    control-plane   7h58m   v1.28.2
              node/zeta-rottigni-tech      Ready    &lt;none&gt;          7h57m   v1.28.2
  </code></pre>
  <p>One strange thing is that my ingress has IP 149.62.189.139, which is my worker node IP (not control plane IP). Is this the correct behavior?</p>
  <p>Despite this, I'm unable to reach my services from the public DNS through my nginx ingress controller. I can, however, correctly reach both services using NodePort from my control plane and worker node SSH sessions, eliminating any application errors.</p>
  <p>It appears that my ingress isn't facing the internet on the correct IP. I also checked my ingress-nginx-controller logs, which seem fine:</p>
  <pre><code>-------------------------------------------------------------------------------
  NGINX Ingress controller
    Release:       v1.9.5
    Build:         f503c4bb5fa7d857ad29e94970eb550c2bc00b7c
    Repository:    https://github.com/kubernetes/ingress-nginx
    nginx version: nginx/1.21.6
  
  -------------------------------------------------------------------------------
  
  W0105 14:11:47.972914       7 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
  I0105 14:11:47.973187       7 main.go:205] &quot;Creating API client&quot; host=&quot;https://10.96.0.1:443&quot;
  I0105 14:11:47.983447       7 main.go:249] &quot;Running in Kubernetes cluster&quot; major=&quot;1&quot; minor=&quot;28&quot; git=&quot;v1.28.5&quot; state=&quot;clean&quot; commit=&quot;506050d61cf291218dfbd41ac93913945c9aa0da&quot; platform=&quot;linux/amd64&quot;
  I0105 14:11:48.470371       7 main.go:101] &quot;SSL fake certificate created&quot; file=&quot;/etc/ingress-controller/ssl/default-fake-certificate.pem&quot;
  I0105 14:11:48.511293       7 ssl.go:536] &quot;loading tls certificate&quot; path=&quot;/usr/local/certificates/cert&quot; key=&quot;/usr/local/certificates/key&quot;
  I0105 14:11:48.529226       7 nginx.go:260] &quot;Starting NGINX Ingress controller&quot;
  I0105 14:11:48.539963       7 event.go:298] Event(v1.ObjectReference{Kind:&quot;ConfigMap&quot;, Namespace:&quot;ingress-nginx&quot;, Name:&quot;ingress-nginx-controller&quot;, UID:&quot;a5edeb27-12a0-4576-ab70-19d8cffb4dfe&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;338&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
  I0105 14:11:49.635029       7 store.go:440] &quot;Found valid IngressClass&quot; ingress=&quot;lr-labs/lr-labs-ingress&quot; ingressclass=&quot;nginx-ingress-class&quot;
  I0105 14:11:49.636090       7 event.go:298] Event(v1.ObjectReference{Kind:&quot;Ingress&quot;, Namespace:&quot;lr-labs&quot;, Name:&quot;lr-labs-ingress&quot;, UID:&quot;5663167b-9b94-4cb6-8670-5e5e1076bfaa&quot;, APIVersion:&quot;networking.k8s.io/v1&quot;, ResourceVersion:&quot;1159&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'Sync' Scheduled for sync
  I0105 14:11:49.638242       7 backend_ssl.go:67] &quot;Adding secret to local store&quot; name=&quot;lr-labs/registry-ssl&quot;
  I0105 14:11:49.641278       7 backend_ssl.go:67] &quot;Adding secret to local store&quot; name=&quot;lr-labs/sandbox-ssl&quot;
  I0105 14:11:49.731218       7 nginx.go:303] &quot;Starting NGINX process&quot;
  I0105 14:11:49.731681       7 leaderelection.go:245] attempting to acquire leader lease ingress-nginx/ingress-nginx-leader...
  I0105 14:11:49.732611       7 nginx.go:323] &quot;Starting validation webhook&quot; address=&quot;:8443&quot; certPath=&quot;/usr/local/certificates/cert&quot; keyPath=&quot;/usr/local/certificates/key&quot;
  I0105 14:11:49.733509       7 controller.go:190] &quot;Configuration changes detected, backend reload required&quot;
  I0105 14:11:49.736538       7 status.go:84] &quot;New leader elected&quot; identity=&quot;ingress-nginx-controller-685dff585b-2s8jp&quot;
  I0105 14:11:49.849528       7 controller.go:210] &quot;Backend successfully reloaded&quot;
  I0105 14:11:49.849813       7 controller.go:221] &quot;Initial sync, sleeping for 1 second&quot;
  I0105 14:11:49.850019       7 event.go:298] Event(v1.ObjectReference{Kind:&quot;Pod&quot;, Namespace:&quot;ingress-nginx&quot;, Name:&quot;ingress-nginx-controller-86968d59df-vdv9f&quot;, UID:&quot;7ac8215f-175f-4496-90a1-dd63d6acad4e&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;6819&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
  I0105 14:12:41.662980       7 status.go:84] &quot;New leader elected&quot; identity=&quot;ingress-nginx-controller-86968d59df-vdv9f&quot;
  I0105 14:12:41.663007       7 leaderelection.go:255] successfully acquired lease ingress-nginx/ingress-nginx-leader
  10.32.0.1 - - [05/Jan/2024:14:36:35 +0000] &quot;GET / HTTP/1.1&quot; 400 248 &quot;-&quot; &quot;curl/7.81.0&quot; 81 0.000 [] [] - - - - 479859864f23ba0523fb9c29c1932d56
  10.32.0.1 - - [05/Jan/2024:14:38:30 +0000] &quot;GET / HTTP/1.1&quot; 308 164 &quot;-&quot; &quot;curl/7.81.0&quot; 85 0.000 [lr-labs-sandbox-rottigni-80] [] - - - - e262c16d9495ef00f412f0ba73dbb453
  10.32.0.1 - - [05/Jan/2024:15:25:27 +0000] &quot;GET / HTTP/1.1&quot; 308 164 &quot;-&quot; &quot;curl/7.81.0&quot; 85 0.000 [lr-labs-sandbox-rottigni-80] [] - - - - 9b662b33bb59be1846863af658f251ab
  10.32.0.1 - - [05/Jan/2024:16:08:32 +0000] &quot;GET / HTTP/1.1&quot; 308 164 &quot;-&quot; &quot;curl/7.81.0&quot; 85 0.000 [lr-labs-sandbox-rottigni-80] [] - - - - a54eda51703800c9a1726a9e1187988d
  </code></pre>
  <p>I even see some manually initiated curl requests for debugging directly to the nginx service using the Host HTTP header.</p>
  <p>Additionally here my working curl commands to my cluster service resources:</p>
  <pre><code>root@epsilon-rottigni-tech:~/k8s.rottigni.tech/k8s# kubectl get svc -A
  NAMESPACE       NAME                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
  default         kubernetes                           ClusterIP   10.96.0.1        &lt;none&gt;        443/TCP                      8h
  ingress-nginx   ingress-nginx-controller             NodePort    10.108.71.135    &lt;none&gt;        80:32095/TCP,443:30221/TCP   8h
  ingress-nginx   ingress-nginx-controller-admission   ClusterIP   10.107.203.218   &lt;none&gt;        443/TCP                      8h
  kube-system     kube-dns                             ClusterIP   10.96.0.10       &lt;none&gt;        53/UDP,53/TCP,9153/TCP       8h
  lr-labs         registry-rottigni                    NodePort    10.106.231.17    &lt;none&gt;        5000:31673/TCP               8h
  lr-labs         sandbox-rottigni                     NodePort    10.111.173.239   &lt;none&gt;        80:31607/TCP                 8h
  root@epsilon-rottigni-tech:~/k8s.rottigni.tech/k8s# curl 10.106.231.17:5000
  root@epsilon-rottigni-tech:~/k8s.rottigni.tech/k8s# curl 10.108.71.135:80
  &lt;html&gt;
  &lt;head&gt;&lt;title&gt;404 Not Found&lt;/title&gt;&lt;/head&gt;
  &lt;body&gt;
  &lt;center&gt;&lt;h1&gt;404 Not Found&lt;/h1&gt;&lt;/center&gt;
  &lt;hr&gt;&lt;center&gt;nginx&lt;/center&gt;
  &lt;/body&gt;
  &lt;/html&gt;
  root@epsilon-rottigni-tech:~/k8s.rottigni.tech/k8s# curl 10.108.71.135:443
  &lt;html&gt;
  &lt;head&gt;&lt;title&gt;400 The plain HTTP request was sent to HTTPS port&lt;/title&gt;&lt;/head&gt;
  &lt;body&gt;
  &lt;center&gt;&lt;h1&gt;400 Bad Request&lt;/h1&gt;&lt;/center&gt;
  &lt;center&gt;The plain HTTP request was sent to HTTPS port&lt;/center&gt;
  &lt;hr&gt;&lt;center&gt;nginx&lt;/center&gt;
  &lt;/body&gt;
  &lt;/html&gt;
  root@epsilon-rottigni-tech:~/k8s.rottigni.tech/k8s# curl 10.111.173.239:80
  &lt;html&gt;
    &lt;head&gt;
      &lt;style&gt;
        html {
          background: url(./Cheddar.jpg) no-repeat center center fixed;
          -webkit-background-size: cover;
          -moz-background-size: cover;
          -o-background-size: cover;
          background-size: cover;
        }
  
        h1 {
          font-family: Arial, Helvetica, sans-serif;
          background: rgba(187, 187, 187, 0.5);
          width: 6em;
          padding: 0.5em 1em;
          margin: 1em;
        }
      &lt;/style&gt;
    &lt;/head&gt;
    &lt;body&gt;
      &lt;h1&gt;Cheddar&lt;/h1&gt;
    &lt;/body&gt;
  &lt;/html&gt;root@epsilon-rottigni-tech:~/k8s.rottigni.tech/k8s#
  </code></pre>
  <p>Could someone please help me debug this problem? Am I missing an important step for this kind of production deployment?</p>
  <p>Thank you everybody!</p>
  <hr />
  <p>UPDATE:</p>
  <p>Given the following situation:</p>
  <pre><code>kubectl get svc,ing -A
  NAMESPACE       NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
  default         service/kubernetes                           ClusterIP   10.96.0.1        &lt;none&gt;        443/TCP                      23h
  ingress-nginx   service/ingress-nginx-controller             NodePort    10.108.71.135    &lt;none&gt;        80:32095/TCP,443:30221/TCP   23h
  ingress-nginx   service/ingress-nginx-controller-admission   ClusterIP   10.107.203.218   &lt;none&gt;        443/TCP                      23h
  kube-system     service/kube-dns                             ClusterIP   10.96.0.10       &lt;none&gt;        53/UDP,53/TCP,9153/TCP       23h
  lr-labs         service/registry-rottigni                    NodePort    10.106.231.17    &lt;none&gt;        5000:31673/TCP               23h
  lr-labs         service/sandbox-rottigni                     NodePort    10.111.173.239   &lt;none&gt;        80:31607/TCP                 23h
  
  NAMESPACE   NAME                                        CLASS                 HOSTS                                          ADDRESS          PORTS     AGE
  lr-labs     ingress.networking.k8s.io/lr-labs-ingress   nginx-ingress-class   registry.rottigni.tech,sandbox.rottigni.tech   149.62.189.139   80, 443   23h
  </code></pre>
  <p>I am able to reach nginx from internet like this:</p>
  <ul>
  <li>curl -k <a href="https://sandbox.rottigni.tech:30221" rel="nofollow noreferrer">https://sandbox.rottigni.tech:30221</a></li>
  <li>curl -k <a href="http://sandbox.rottigni.tech:32095" rel="nofollow noreferrer">http://sandbox.rottigni.tech:32095</a></li>
  </ul>
  <p>But I can't like this:</p>
  <ul>
  <li>curl -k <a href="https://sandbox.rottigni.tech" rel="nofollow noreferrer">https://sandbox.rottigni.tech</a></li>
  <li>curl -k <a href="http://sandbox.rottigni.tech" rel="nofollow noreferrer">http://sandbox.rottigni.tech</a></li>
  </ul>
  
Tags: <kubernetes><kubectl><kubeadm><ingress-nginx>
Title: How to deploy an nginx ingress to a kubeadm cluster
UserDisplayName: Lorenzo Rottigni
ThreadText: # How to deploy an nginx ingress to a kubeadm cluster
  
  (Asked by: username_0 on 2024-01-06)
  
  I'm attempting to set up my own bare-metal Kubernetes cluster composed of a control plane (VPS 149.62.186.95) and a worker node (VPS 149.62.189.139). Both my VPS are running Ubuntu 22.04 with 2 vCPUs and 4GB vRAM.
  
  After installing Kubernetes (`kubeadm`) on both, I ran the following commands on my control plane:
  
  ```
  kubeadm init
  
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
  
  # Weave deamonset
  kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml
  
  # baremetal nginx ingress
  kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml
  
  ```
  
  I edited my DNS records as follows:
  
  * registry.username_0.tech -> 149.62.186.95 (control plane)
  * sandbox.username_0.tech -> 149.62.186.95 (control plane)
  
  I generated SSL certs for these domains on my control plane and stored them in Kubernetes secrets.
  
  Then, I applied my changes using kubectl apply -R -f . inside my repository. Here's the most relevant file, ingress.yml:
  
  ```
  apiVersion: networking.k8s.io/v1
  kind: IngressClass
  metadata:
    name: nginx-ingress-class
  spec:
    controller: k8s.io/ingress-nginx
  ---
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: lr-labs-ingress
    namespace: lr-labs
    annotations:
      # cert-manager.io/issuer: "letsencrypt-prod"
      nginx.ingress.kubernetes.io/rewrite-target: /$1
  spec:
    ingressClassName: nginx-ingress-class
    tls:
      - hosts:
          - registry.username_0.tech
        secretName: registry-ssl
      - hosts:
          - sandbox.username_0.tech
        secretName: sandbox-ssl
  
    rules:
      - host: registry.username_0.tech
        http:
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: registry-username_0
                  port:
                    number: 5000
  
      - host: sandbox.username_0.tech
        http:
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: sandbox-username_0
                  port:
                    number: 80
  
  ```
  
  At this point, everything seems to be working on the kubectl side, according to the following output:
  
  ```
  Every 2.0s: kubectl get pods,deploy,svc,ing,endpoints,node -A                                                                                                                                                                           
  
  NAMESPACE       NAME                                                READY   STATUS      RESTARTS        AGE
  ingress-nginx   pod/ingress-nginx-admission-create-wn7xr            0/1     Completed   0               7h58m
  ingress-nginx   pod/ingress-nginx-admission-patch-dtzxf             0/1     Completed   0               7h58m
  ingress-nginx   pod/ingress-nginx-controller-86968d59df-vdv9f       1/1     Running     0               6h52m
  kube-system     pod/coredns-5dd5756b68-h4jwb                        1/1     Running     0               7h58m
  kube-system     pod/coredns-5dd5756b68-jrsdq                        1/1     Running     0               7h58m
  kube-system     pod/etcd-epsilon-username_0-tech                      1/1     Running     0               7h58m
  kube-system     pod/kube-apiserver-epsilon-username_0-tech            1/1     Running     0               7h58m
  kube-system     pod/kube-controller-manager-epsilon-username_0-tech   1/1     Running     0               7h58m
  kube-system     pod/kube-proxy-4v8bc                                1/1     Running     0               7h58m
  kube-system     pod/kube-proxy-8hpjj                                1/1     Running     0               7h57m
  kube-system     pod/kube-scheduler-epsilon-username_0-tech            1/1     Running     0               7h58m
  kube-system     pod/weave-net-gvzk5                                 2/2     Running     1 (7h58m ago)   7h58m
  kube-system     pod/weave-net-tbr92                                 2/2     Running     0               7h57m
  lr-labs         pod/registry-username_0-5c6cfd877f-qqxsw              1/1     Running     0               7h54m
  lr-labs         pod/sandbox-username_0-9668f7447-dv7js                1/1     Running     0               7h54m
  
  NAMESPACE       NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
  ingress-nginx   deployment.apps/ingress-nginx-controller   1/1     1            1           7h58m
  kube-system     deployment.apps/coredns                    2/2     2            2           7h58m
  lr-labs         deployment.apps/registry-username_0          1/1     1            1           7h54m
  lr-labs         deployment.apps/sandbox-username_0           1/1     1            1           7h54m
  
  NAMESPACE       NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
  default         service/kubernetes                           ClusterIP   10.96.0.1        <none>        443/TCP                      7h58m
  ingress-nginx   service/ingress-nginx-controller             NodePort    10.108.71.135    <none>        80:32095/TCP,443:30221/TCP   7h58m
  ingress-nginx   service/ingress-nginx-controller-admission   ClusterIP   10.107.203.218   <none>        443/TCP                      7h58m
  kube-system     service/kube-dns                             ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP       7h58m
  lr-labs         service/registry-username_0                    NodePort    10.106.231.17    <none>        5000:31673/TCP               7h54m
  lr-labs         service/sandbox-username_0                     NodePort    10.111.173.239   <none>        80:31607/TCP                 7h54m
  
  NAMESPACE   NAME                                        CLASS                 HOSTS                                          ADDRESS          PORTS     AGE
  lr-labs     ingress.networking.k8s.io/lr-labs-ingress   nginx-ingress-class   registry.username_0.tech,sandbox.username_0.tech   149.62.189.139   80, 443   7h54m
  
  NAMESPACE       NAME                                           ENDPOINTS                                            AGE
  default         endpoints/kubernetes                           149.62.186.95:6443                                   7h58m
  ingress-nginx   endpoints/ingress-nginx-controller             10.44.0.2:443,10.44.0.2:80                           7h58m
  ingress-nginx   endpoints/ingress-nginx-controller-admission   10.44.0.2:8443                                       7h58m
  kube-system     endpoints/kube-dns                             10.32.0.2:53,10.32.0.3:53,10.32.0.2:53 + 3 more...   7h58m
  lr-labs         endpoints/registry-username_0                    10.44.0.3:5000                                       7h54m
  lr-labs         endpoints/sandbox-username_0                     10.44.0.1:80                                         7h54m
  
  NAMESPACE   NAME                         STATUS   ROLES           AGE     VERSION
              node/epsilon-username_0-tech   Ready    control-plane   7h58m   v1.28.2
              node/zeta-username_0-tech      Ready    <none>          7h57m   v1.28.2
  
  ```
  
  One strange thing is that my ingress has IP 149.62.189.139, which is my worker node IP (not control plane IP). Is this the correct behavior?
  
  Despite this, I'm unable to reach my services from the public DNS through my nginx ingress controller. I can, however, correctly reach both services using NodePort from my control plane and worker node SSH sessions, eliminating any application errors.
  
  It appears that my ingress isn't facing the internet on the correct IP. I also checked my ingress-nginx-controller logs, which seem fine:
  
  ```
  -------------------------------------------------------------------------------
  NGINX Ingress controller
    Release:       v1.9.5
    Build:         f503c4bb5fa7d857ad29e94970eb550c2bc00b7c
    Repository:    https://github.com/kubernetes/ingress-nginx
    nginx version: nginx/1.21.6
  
  -------------------------------------------------------------------------------
  
  W0105 14:11:47.972914       7 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
  I0105 14:11:47.973187       7 main.go:205] "Creating API client" host="https://10.96.0.1:443"
  I0105 14:11:47.983447       7 main.go:249] "Running in Kubernetes cluster" major="1" minor="28" git="v1.28.5" state="clean" commit="506050d61cf291218dfbd41ac93913945c9aa0da" platform="linux/amd64"
  I0105 14:11:48.470371       7 main.go:101] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
  I0105 14:11:48.511293       7 ssl.go:536] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
  I0105 14:11:48.529226       7 nginx.go:260] "Starting NGINX Ingress controller"
  I0105 14:11:48.539963       7 event.go:298] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"a5edeb27-12a0-4576-ab70-19d8cffb4dfe", APIVersion:"v1", ResourceVersion:"338", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
  I0105 14:11:49.635029       7 store.go:440] "Found valid IngressClass" ingress="lr-labs/lr-labs-ingress" ingressclass="nginx-ingress-class"
  I0105 14:11:49.636090       7 event.go:298] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"lr-labs", Name:"lr-labs-ingress", UID:"5663167b-9b94-4cb6-8670-5e5e1076bfaa", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1159", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
  I0105 14:11:49.638242       7 backend_ssl.go:67] "Adding secret to local store" name="lr-labs/registry-ssl"
  I0105 14:11:49.641278       7 backend_ssl.go:67] "Adding secret to local store" name="lr-labs/sandbox-ssl"
  I0105 14:11:49.731218       7 nginx.go:303] "Starting NGINX process"
  I0105 14:11:49.731681       7 leaderelection.go:245] attempting to acquire leader lease ingress-nginx/ingress-nginx-leader...
  I0105 14:11:49.732611       7 nginx.go:323] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
  I0105 14:11:49.733509       7 controller.go:190] "Configuration changes detected, backend reload required"
  I0105 14:11:49.736538       7 status.go:84] "New leader elected" identity="ingress-nginx-controller-685dff585b-2s8jp"
  I0105 14:11:49.849528       7 controller.go:210] "Backend successfully reloaded"
  I0105 14:11:49.849813       7 controller.go:221] "Initial sync, sleeping for 1 second"
  I0105 14:11:49.850019       7 event.go:298] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-86968d59df-vdv9f", UID:"7ac8215f-175f-4496-90a1-dd63d6acad4e", APIVersion:"v1", ResourceVersion:"6819", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
  I0105 14:12:41.662980       7 status.go:84] "New leader elected" identity="ingress-nginx-controller-86968d59df-vdv9f"
  I0105 14:12:41.663007       7 leaderelection.go:255] successfully acquired lease ingress-nginx/ingress-nginx-leader
  10.32.0.1 - - [05/Jan/2024:14:36:35 +0000] "GET / HTTP/1.1" 400 248 "-" "curl/7.81.0" 81 0.000 [] [] - - - - 479859864f23ba0523fb9c29c1932d56
  10.32.0.1 - - [05/Jan/2024:14:38:30 +0000] "GET / HTTP/1.1" 308 164 "-" "curl/7.81.0" 85 0.000 [lr-labs-sandbox-username_0-80] [] - - - - e262c16d9495ef00f412f0ba73dbb453
  10.32.0.1 - - [05/Jan/2024:15:25:27 +0000] "GET / HTTP/1.1" 308 164 "-" "curl/7.81.0" 85 0.000 [lr-labs-sandbox-username_0-80] [] - - - - 9b662b33bb59be1846863af658f251ab
  10.32.0.1 - - [05/Jan/2024:16:08:32 +0000] "GET / HTTP/1.1" 308 164 "-" "curl/7.81.0" 85 0.000 [lr-labs-sandbox-username_0-80] [] - - - - a54eda51703800c9a1726a9e1187988d
  
  ```
  
  I even see some manually initiated curl requests for debugging directly to the nginx service using the Host HTTP header.
  
  Additionally here my working curl commands to my cluster service resources:
  
  ```
  root@epsilon-username_0-tech:~/k8s.username_0.tech/k8s# kubectl get svc -A
  NAMESPACE       NAME                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
  default         kubernetes                           ClusterIP   10.96.0.1        <none>        443/TCP                      8h
  ingress-nginx   ingress-nginx-controller             NodePort    10.108.71.135    <none>        80:32095/TCP,443:30221/TCP   8h
  ingress-nginx   ingress-nginx-controller-admission   ClusterIP   10.107.203.218   <none>        443/TCP                      8h
  kube-system     kube-dns                             ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP       8h
  lr-labs         registry-username_0                    NodePort    10.106.231.17    <none>        5000:31673/TCP               8h
  lr-labs         sandbox-username_0                     NodePort    10.111.173.239   <none>        80:31607/TCP                 8h
  root@epsilon-username_0-tech:~/k8s.username_0.tech/k8s# curl 10.106.231.17:5000
  root@epsilon-username_0-tech:~/k8s.username_0.tech/k8s# curl 10.108.71.135:80
  <html>
  <head><title>404 Not Found</title></head>
  <body>
  <center><h1>404 Not Found</h1></center>
  <hr><center>nginx</center>
  </body>
  </html>
  root@epsilon-username_0-tech:~/k8s.username_0.tech/k8s# curl 10.108.71.135:443
  <html>
  <head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
  <body>
  <center><h1>400 Bad Request</h1></center>
  <center>The plain HTTP request was sent to HTTPS port</center>
  <hr><center>nginx</center>
  </body>
  </html>
  root@epsilon-username_0-tech:~/k8s.username_0.tech/k8s# curl 10.111.173.239:80
  <html>
    <head>
      <style>
        html {
          background: url(./Cheddar.jpg) no-repeat center center fixed;
          -webkit-background-size: cover;
          -moz-background-size: cover;
          -o-background-size: cover;
          background-size: cover;
        }
  
        h1 {
          font-family: Arial, Helvetica, sans-serif;
          background: rgba(187, 187, 187, 0.5);
          width: 6em;
          padding: 0.5em 1em;
          margin: 1em;
        }
      </style>
    </head>
    <body>
      <h1>Cheddar</h1>
    </body>
  </html>root@epsilon-username_0-tech:~/k8s.username_0.tech/k8s#
  
  ```
  
  Could someone please help me debug this problem? Am I missing an important step for this kind of production deployment?
  
  Thank you everybody!
  
  ---
  
  UPDATE:
  
  Given the following situation:
  
  ```
  kubectl get svc,ing -A
  NAMESPACE       NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
  default         service/kubernetes                           ClusterIP   10.96.0.1        <none>        443/TCP                      23h
  ingress-nginx   service/ingress-nginx-controller             NodePort    10.108.71.135    <none>        80:32095/TCP,443:30221/TCP   23h
  ingress-nginx   service/ingress-nginx-controller-admission   ClusterIP   10.107.203.218   <none>        443/TCP                      23h
  kube-system     service/kube-dns                             ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP       23h
  lr-labs         service/registry-username_0                    NodePort    10.106.231.17    <none>        5000:31673/TCP               23h
  lr-labs         service/sandbox-username_0                     NodePort    10.111.173.239   <none>        80:31607/TCP                 23h
  
  NAMESPACE   NAME                                        CLASS                 HOSTS                                          ADDRESS          PORTS     AGE
  lr-labs     ingress.networking.k8s.io/lr-labs-ingress   nginx-ingress-class   registry.username_0.tech,sandbox.username_0.tech   149.62.189.139   80, 443   23h
  
  ```
  
  I am able to reach nginx from internet like this:
  
  * curl -k <https://sandbox.username_0.tech:30221>
  * curl -k <http://sandbox.username_0.tech:32095>
  
  But I can't like this:
  
  * curl -k <https://sandbox.username_0.tech>
  * curl -k <http://sandbox.username_0.tech>
  
  **Tags:** kubernetes, kubectl, kubeadm, ingress-nginx
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-07. Score: 0
  
  Basically you did everything correctly.
  The chosen way to make your service reachable from the outside does not allow you to use specific, low, ports in order to keep isolation between services.
  You have two options:
  
  1. use a service of type LoadBalancer which will allocate a separate IP for the service, providing isolation between services and allowing you a specific port like 80 or 443
  2. modify the ingress deployment and use a `hostPort` for the nginx containers, you can then use the node IPs where a pod is running. This can lead to a lot of headaches (f.e. when updating the deployment) but might be an easy solution to get started
  
  > **Comments:**
  > **username_0**: Thanks for you answer, I fixed my problem using MetalLB with a super basic config. I misunderstood the ingress resource behaviour because of his IP address… I still don’t understand why the ingress resource has an IP assigned since its only purpose is to provide an nginx configuration to the ingress-nginx-controller service. Everything is now working as expected, thanks anyway for your help!  
  > **username_1**: If you are using MetalLB and It is configured correctly it should provide you with a dedicated IP as it is a LoadBalancer implementation. Please accept the answer if it solved your issue.  
  
  ---
  
  