Answers: <p>I figured it out, and  SELinux is the difference.  That is, SELinux on the HAProxy host (not a cluster node):</p>
  <p>&quot;SELinux only allows the web server to make outbound connections to a limited set of ports&quot;</p>
  <p>That is, you can't make an outbound http request to any port in the NodePort range (30000-32768) without opening that port on the &quot;client&quot;, which is the HAProxy server in this case.</p>
  <pre><code>sudo semanage port --add --type http_port_t --proto tcp 30080
  </code></pre>
  <p>More specifically, I guess, the security context that HAProxy runs under is different than curl.</p>
  
Answers: CC BY-SA 4.0
Answers: 2021-08-31T03:59:09.747Z
Answers: 2021-08-31T22:15:05.660Z
Answers: 2021-08-31T22:15:05.660Z
Answers: Adam Winter
Body: <p>I have a host running HAProxy already.  It's been up and running since before I did anything with Kubernetes.  It works flawlessly as a reverse proxy and SSL terminator for various web things in various Docker containers on various other host machines.</p>
  <p>Now I have a Kubernetes cluster up and running across some of those other machines.  I've created the NodePort Service that exposes port 30080 on each worker node, as follows:</p>
  <pre><code>apiVersion: v1
  kind: Service
  metadata:
    name: snginx
    labels:
      app: nginx
  spec:
    type: NodePort
    externalTrafficPolicy: Local    #Cluster or Local
    selector:
      app: nginx
    ports:
    - protocol: TCP
      port: 8080
      targetPort: 80
      nodePort: 30080
  </code></pre>
  <p>From the machine running HAProxy (which is not part of the cluster), I can curl the NodePort successfully  ( curl 10.0.0.20:30080 ), and I get &quot;Welcome to nginx!...&quot;   However, if I set that NodePort as a backend in HAProxy, I get a 503 &quot;No server is available&quot;, and HAProxy traffic log says:</p>
  <pre><code>localhost haproxy[18489]: [redactedIP]:49399 [30/Aug/2021:19:24:00.764] http-in~ load/load 66/0/-1/-1/67 503 212 - - SC-- 1/1/0/0/3 0/0 &quot;GET / HTTP/1.1&quot;
  </code></pre>
  <p>The haproxy admin log says:</p>
  <pre><code>Aug 30 20:07:13 localhost haproxy[18839]: Server load/load is DOWN, reason: Layer4 connection problem, info: &quot;General socket error (Permission denied)&quot;
  </code></pre>
  <p>However, I've disabled the firewall with</p>
  <pre><code>sudo systemctl disable --now firewalld
  </code></pre>
  <p>and verified the status is not running.  Also, SELinux was disabled when I installed the cluster.  Also, I can ping 10.0.0.20 just fine.</p>
  <p>&quot;load&quot; is the hostname I'm using for testing load balancing (i.e.  load.mydomain.com).</p>
  <p>Also, if I use static NAT on my physical router to route directly to that NodePort, from outside the building, it works as expected.</p>
  <p>What gives?  What's the difference between the proxied request and curl?</p>
  <p>Thank you.</p>
  
Tags: <kubernetes><load-balance><kubectl>
Title: HAProxy Not Working with Kubernetes NodePort for Backend (Bare Metal)
UserDisplayName: Adam Winter
ThreadText: # HAProxy Not Working with Kubernetes NodePort for Backend (Bare Metal)
  
  (Asked by: username_0 on 2021-08-31)
  
  I have a host running HAProxy already. It's been up and running since before I did anything with Kubernetes. It works flawlessly as a reverse proxy and SSL terminator for various web things in various Docker containers on various other host machines.
  
  Now I have a Kubernetes cluster up and running across some of those other machines. I've created the NodePort Service that exposes port 30080 on each worker node, as follows:
  
  ```
  apiVersion: v1
  kind: Service
  metadata:
    name: snginx
    labels:
      app: nginx
  spec:
    type: NodePort
    externalTrafficPolicy: Local    #Cluster or Local
    selector:
      app: nginx
    ports:
    - protocol: TCP
      port: 8080
      targetPort: 80
      nodePort: 30080
  
  ```
  
  From the machine running HAProxy (which is not part of the cluster), I can curl the NodePort successfully ( curl 10.0.0.20:30080 ), and I get "Welcome to nginx!..." However, if I set that NodePort as a backend in HAProxy, I get a 503 "No server is available", and HAProxy traffic log says:
  
  ```
  localhost haproxy[18489]: [redactedIP]:49399 [30/Aug/2021:19:24:00.764] http-in~ load/load 66/0/-1/-1/67 503 212 - - SC-- 1/1/0/0/3 0/0 "GET / HTTP/1.1"
  
  ```
  
  The haproxy admin log says:
  
  ```
  Aug 30 20:07:13 localhost haproxy[18839]: Server load/load is DOWN, reason: Layer4 connection problem, info: "General socket error (Permission denied)"
  
  ```
  
  However, I've disabled the firewall with
  
  ```
  sudo systemctl disable --now firewalld
  
  ```
  
  and verified the status is not running. Also, SELinux was disabled when I installed the cluster. Also, I can ping 10.0.0.20 just fine.
  
  "load" is the hostname I'm using for testing load balancing (i.e. load.mydomain.com).
  
  Also, if I use static NAT on my physical router to route directly to that NodePort, from outside the building, it works as expected.
  
  What gives? What's the difference between the proxied request and curl?
  
  Thank you.
  
  **Tags:** kubernetes, load-balance, kubectl
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-08-31. Score: 1
  
  I figured it out, and SELinux is the difference. That is, SELinux on the HAProxy host (not a cluster node):
  
  "SELinux only allows the web server to make outbound connections to a limited set of ports"
  
  That is, you can't make an outbound http request to any port in the NodePort range (30000-32768) without opening that port on the "client", which is the HAProxy server in this case.
  
  ```
  sudo semanage port --add --type http_port_t --proto tcp 30080
  
  ```
  
  More specifically, I guess, the security context that HAProxy runs under is different than curl.
  
  
  ---
  
  