Body: <p>We have a development kubernetes cluster (hosted in EKS) to which engineers push feature branches. Each feature branch creates the same set of Deployments and Services in its own isolated namespace. We have dynamic load balancing for the HTTP services using nginx ingress, allowing us to create hostnames on-demand based on the feature branch name and expose the HTTP services consistently on ports 80 and/or 443. This creates a single ELB and can do host based routing.</p>
  <p>However, we also have some non-HTTP services that we want to be able to expose on a consistent port across namespaces. Lets just say port 4000 for the sake of a concrete example. In addition we'd like to have friendly human readable hostnames for these services.</p>
  <p>It seems like the hostname problem can be solved with ExternalDNS. However, I can't find a way to consistently expose port 4000 without incurring an additional load balancer for every namespace. i.e. if I have 100 namespaces, I'll have 100 load balancers.</p>
  <p>I feel like I understand the networking sufficiently well and this is simply a hard limitation, I would love to be proved wrong. The protocols I'm dealing with don't send the hostname that was requested so I have to assign a unique IP address for each service (assuming the port can't change). I guess what I'm looking for is if anyone has a more cost efficient way of doing this, because incurring the load balancer costs aren't acceptable.</p>
  <p>Before anyone suggests using different ports across the namespaces, yes it would solve the problem, but it makes the development experience more frustrating, so I'd like to avoid it. Also, <code>kubectl port-forward</code> works, it's what we've been using, I'm investigating whether we can improve on that so engineers can access the services immediately after deploying a namespace without having to run any extra commands first.</p>
  <p>Finally some additional context, I don't really need load balancing. This is a dev cluster and every Service resource will only ever resolve to a single Pod. I don't know if that provides any unique workarounds.</p>
  
Comments: CC BY-SA 4.0
Comments: 2023-06-25T12:11:41.410Z
Comments: "non-HTTP services", can you elaborate a bit more?
Comments: Kyslik
Comments: CC BY-SA 4.0
Comments: 2023-06-26T16:08:45.407Z
Comments: Redis, PostgreSQL (not it's HTTP APIs), AMQP, possibly other things. Right now if an engineer wants to say, get a psql shell to their dev DB, they have to exec into a container in the cluster and run psql from there. I'd like them to be able to psql directly from their laptop, or say, run a test locally that can leverage the dev database they have in the cluster.
  
  We've tried telepresence as a solution but it was extremely slow an unreliable.
Comments: Endophage
Comments: CC BY-SA 4.0
Comments: 2023-06-28T15:09:32.107Z
Comments: Developers need to interact with these TCP services while developing; right? What happens after they are are done with their "branch/work"?
Comments: Kyslik
Comments: CC BY-SA 4.0
Comments: 2023-06-28T16:11:54.857Z
Comments: The namespace is torn down once it's no longer required, but I don't see how that's relevant to the networking issue I'm trying to solve.
Comments: Endophage
Tags: <kubernetes><networking><eks>
Title: Expose consistent port across many namespaces in a dev k8s cluster
UserDisplayName: Endophage
ThreadText: # Expose consistent port across many namespaces in a dev k8s cluster
  
  (Asked by: username_0 on 2023-04-21)
  
  We have a development kubernetes cluster (hosted in EKS) to which engineers push feature branches. Each feature branch creates the same set of Deployments and Services in its own isolated namespace. We have dynamic load balancing for the HTTP services using nginx ingress, allowing us to create hostnames on-demand based on the feature branch name and expose the HTTP services consistently on ports 80 and/or 443. This creates a single ELB and can do host based routing.
  
  However, we also have some non-HTTP services that we want to be able to expose on a consistent port across namespaces. Lets just say port 4000 for the sake of a concrete example. In addition we'd like to have friendly human readable hostnames for these services.
  
  It seems like the hostname problem can be solved with ExternalDNS. However, I can't find a way to consistently expose port 4000 without incurring an additional load balancer for every namespace. i.e. if I have 100 namespaces, I'll have 100 load balancers.
  
  I feel like I understand the networking sufficiently well and this is simply a hard limitation, I would love to be proved wrong. The protocols I'm dealing with don't send the hostname that was requested so I have to assign a unique IP address for each service (assuming the port can't change). I guess what I'm looking for is if anyone has a more cost efficient way of doing this, because incurring the load balancer costs aren't acceptable.
  
  Before anyone suggests using different ports across the namespaces, yes it would solve the problem, but it makes the development experience more frustrating, so I'd like to avoid it. Also, `kubectl port-forward` works, it's what we've been using, I'm investigating whether we can improve on that so engineers can access the services immediately after deploying a namespace without having to run any extra commands first.
  
  Finally some additional context, I don't really need load balancing. This is a dev cluster and every Service resource will only ever resolve to a single Pod. I don't know if that provides any unique workarounds.
  
  **Tags:** kubernetes, networking, eks
  
  > **Comments:**
  > **username_1**: "non-HTTP services", can you elaborate a bit more?  
  > **username_0**: Redis, PostgreSQL (not it's HTTP APIs), AMQP, possibly other things. Right now if an engineer wants to say, get a psql shell to their dev DB, they have to exec into a container in the cluster and run psql from there. I'd like them to be able to psql directly from their laptop, or say, run a test locally that can leverage the dev database they have in the cluster.  We've tried telepresence as a solution but it was extremely slow an unreliable.  
  > **username_1**: Developers need to interact with these TCP services while developing; right? What happens after they are are done with their "branch/work"?  
  > **username_0**: The namespace is torn down once it's no longer required, but I don't see how that's relevant to the networking issue I'm trying to solve.  
  
  ---
  
  ## 0 answers
  
  