Answers: <p>Yes, you can. <code>HAProxy</code> is purpose built to balance <em>any</em> TCP or HTTP connection. My old company used to run every component of Kafka with HAproxy.</p>
  
  <p>Here is a whole project dedicated to running auto-scaling Kafka in Docker using HAproxy: <a href="https://github.com/pozgo/docker-kafka" rel="nofollow noreferrer">https://github.com/pozgo/docker-kafka</a></p>
  
  <p>You can even see their HAproxy config here where they specify the broker check (port 9092): <a href="https://github.com/pozgo/docker-kafka/blob/master/haproxy.cfg" rel="nofollow noreferrer">https://github.com/pozgo/docker-kafka/blob/master/haproxy.cfg</a></p>
  
Answers: CC BY-SA 3.0
Answers: 2018-02-23T16:56:27.197Z
Answers: Thank you @BoomShadow... I am pretty familiar with HAProxy. See my edit for a bit of clarification. I am strongly pondering to delete the question because I'm not happy with it - I think I need to learn more about Kafka to express what I really mean (basically, how to proceed when the pod of a single broker is scaled up, how they then manage to synchronize their data - both reading and writing, and so on). But meanwhile, your answer might be useful for others.
Answers: AnoE
Answers: CC BY-SA 3.0
Answers: 2018-02-23T19:50:00.717Z
Answers: Ah, I gotcha. Your question would actually be more of "how to do" rather than "can one do". I wish I could help more there. I don't have access to the custom config files used by my old team, or else I would share snippets here with you. I personally hadn't touched that part of the infrastructure; I just knew it was there and it's basic arrangement. I think you should leave your question up and see if anyone else can jump in and offer solid insight of what you're looking for. The edit you made clarified things well.
Answers: BoomShadow
Answers: CC BY-SA 3.0
Answers: 2018-02-23T15:50:04.320Z
Answers: 2018-02-23T15:50:04.320Z
Answers: BoomShadow
Body: <p>Is it possible to have several brokers behind a single TCP/IP port? </p>
  
  <p>I.e., offer a single endpoint for Kafka clients, and having a scalable amount of brokers behind that?</p>
  
  <p>In my company, some people are configuring a (scaling) Kafka server for us, and they insist that this is not possible, and that we need to configure every single broker as a separate endpoint...</p>
  
  <p>This is in a OpenShift context; it would be nice to just be able to scale the Kafka pods up and down as needed, while using the usual loadbalancing to offer a single port to the outside world.</p>
  
  <p><strong><em>EDIT:</em></strong> I am familiar with HAProxy; the question is about Kafka, not OpenShift/ HAProxy. Is there anything special to be configured (in the Kafka broker image)</p>
  
  <ul>
  <li>Are they all synchronized amongst each other?</li>
  <li>Are there examples for that somewhere?</li>
  </ul>
  
  <p>Googling for "OpenShift Apache Kafka" etc. brings up plenty of examples, but all that I found seem to have a different service for each Apache Kafka broker (i.e., explicitly running 3 separate Kafka deployments inside OpenShift, with separate OpenShift services, one each). This is what I wish to avoid. I want to horizontally scale the number of Kafka processes behind HAProxy, and have one single external IP. </p>
  
Tags: <openshift><kafka><loadbalancing>
Title: Multiple Kafka brokers behind a single OpenShift service?
UserDisplayName: AnoE
ThreadText: # Multiple Kafka brokers behind a single OpenShift service?
  
  (Asked by: username_0 on 2018-02-23)
  
  Is it possible to have several brokers behind a single TCP/IP port?
  
  I.e., offer a single endpoint for Kafka clients, and having a scalable amount of brokers behind that?
  
  In my company, some people are configuring a (scaling) Kafka server for us, and they insist that this is not possible, and that we need to configure every single broker as a separate endpoint...
  
  This is in a OpenShift context; it would be nice to just be able to scale the Kafka pods up and down as needed, while using the usual loadbalancing to offer a single port to the outside world.
  
  ***EDIT:*** I am familiar with HAProxy; the question is about Kafka, not OpenShift/ HAProxy. Is there anything special to be configured (in the Kafka broker image)
  
  * Are they all synchronized amongst each other?
  * Are there examples for that somewhere?
  
  Googling for "OpenShift Apache Kafka" etc. brings up plenty of examples, but all that I found seem to have a different service for each Apache Kafka broker (i.e., explicitly running 3 separate Kafka deployments inside OpenShift, with separate OpenShift services, one each). This is what I wish to avoid. I want to horizontally scale the number of Kafka processes behind HAProxy, and have one single external IP.
  
  **Tags:** openshift, kafka, loadbalancing
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-02-23. Score: 1
  
  Yes, you can. `HAProxy` is purpose built to balance *any* TCP or HTTP connection. My old company used to run every component of Kafka with HAproxy.
  
  Here is a whole project dedicated to running auto-scaling Kafka in Docker using HAproxy: <https://github.com/pozgo/docker-kafka>
  
  You can even see their HAproxy config here where they specify the broker check (port 9092): <https://github.com/pozgo/docker-kafka/blob/master/haproxy.cfg>
  
  > **Comments:**
  > **username_0**: Thank you @username_1... I am pretty familiar with HAProxy. See my edit for a bit of clarification. I am strongly pondering to delete the question because I'm not happy with it - I think I need to learn more about Kafka to express what I really mean (basically, how to proceed when the pod of a single broker is scaled up, how they then manage to synchronize their data - both reading and writing, and so on). But meanwhile, your answer might be useful for others.  
  > **username_1**: Ah, I gotcha. Your question would actually be more of "how to do" rather than "can one do". I wish I could help more there. I don't have access to the custom config files used by my old team, or else I would share snippets here with you. I personally hadn't touched that part of the infrastructure; I just knew it was there and it's basic arrangement. I think you should leave your question up and see if anyone else can jump in and offer solid insight of what you're looking for. The edit you made clarified things well.  
  
  ---
  
  