title: How to configure Kafka RPC caller topic and group
body: <p>I'm trying to implement an RPC architecture using Kafka as a message broker. The decision of using Kafka instead of another message broker solution is dictated by the current context.</p>
  
  <p>The actual implementation consists on two different types of service:</p>
  
  <ul>
  <li>The <strong>receiver</strong>: this service receives messages from a Kafka topic which consumes, processes the messages and then publish the response message to a response topic;</li>
  <li>The <strong>caller</strong>: this service receives HTTP requests, then publish messages to the receiver topic, consumes the response topic of the receiver service for the response message, then returns it as an HTTP response.</li>
  </ul>
  
  <p>The request/response messages published in the topics are related by the message key.</p>
  
  <p>The <strong>receiver</strong> implementation was fairly simple: at startup, it creates the "request" and "response" topic, then starts consuming the request topic with the service group id (many instances of the receiver will share the same group id in order to implement a proper request balance). When a request arrives, the service processes the request and then publish the response in the response topic.</p>
  
  <p>My problem is with the <strong>caller</strong> implementation, in particular while consuming the response from the response queue.</p>
  
  <p>With the following assumptions:</p>
  
  <ol>
  <li>The HTTP requests must be managed concurrently;</li>
  <li>There could be more than one instance of this caller service.</li>
  </ol>
  
  <p>every single thread/service must receive <em>all the messages</em> in the response topic, in order to find the message with the corresponding request key.</p>
  
  <p>As an example, imagine that two <strong>receiver</strong> services produce two messages with keys <strong>1</strong> and <strong>2</strong> respectively. These messages will be published in the <strong>receiver</strong> topic, and processed. The response will then be published in the topic <strong>receiver-responses</strong>. If the two <strong>receiver</strong> services share the same group-id, it could be that response <strong>1</strong> arrives to the service that published message <strong>2</strong> and vice versa, resulting in a HTTP timeout.</p>
  
  <p>To avoid this problem, I've managed to think these possible solutions:</p>
  
  <ol>
  <li>Creating a new <strong>group</strong> for every request (EDIT: but a group cannot be deleted via code, hence it would be necessary another service to <em>clean</em> the zookeeper from these groups);</li>
  <li>Creating a new <strong>topic</strong> for every request, then delete it afterwards.</li>
  </ol>
  
  <p>Hoping that I made myself sufficiently clear - I must admit I am a beginner to Kafka - my question would be: </p>
  
  <p>Which solution is more costly than the other? Or is there another topic/group configuration that could achieve the assumption <strong>3</strong>?</p>
  
  <p>Thanks.</p>