RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
If you want features of RabbitMQ (specifically queue like behavior) but the scalability of Kafka then you probably want Apache Pulsar.<p>To elaborate on that a bit the main things Pulsar gives you are:<p>1. Still underlying distributed stream based architecture, this is what makes it able to do Kafka like things.<p>2. Broker side management of subscription state which allow out of order acknowledgement, this means you can use it like a queue. (Subscriptions sort of act like AMQP mailboxes but without the exchange routing semantics). Vs Kafka which can only do cumulative acknowledgement, i.e head of line blocking.<p>3. Separated &quot;compute&quot; and storage. By storing data in Bookkeeper you can scale your needs to support a lot of consumers separately from how you stash the data those consumers need to read vs Kafka where these 2 are coupled and an imbalance between the two becomes awkward.<p>4. In built offload with transparent pass-through read. When your data falls off the retention cliff for your standard broker cluster the data can be archived to object storage. The broker can transparently handle read request for these earlier messages though, just with higher startup latency to pull the archived ledgers.<p>5. Way more plugability than Kafka, in-fact similar plugability as RabbitMQ. You can implement your own authz&#x2F;authn, a different listener to support a different protocol (there is a Kafka one, MQTT, AMQP etc).<p>6. Much greater metadata scalability. Before the new KRaft implementation the layout of metadata in ZK meant that you couldn&#x27;t feasibly have more than about 10k topics. Especially because of how long the downtime would be on controller failover. Pulsar can easily support much larger numbers of topics which prevents needing to use a firehose design when you would prefer individual topics per tenant&#x2F;customer&#x2F;whatever.