Kafka Spawns Open-Source KarelDB
I&#x27;ve been looking into Kafka lately, because I have a need for &quot;the thing that Kafka is in practice&quot; (not a message queue per se; rather, an indefinitely-durable event store that supports message-queuing API semantics for consumers.)<p>And, while Kafka <i>does</i> do this, its infrastructure needs are kind of ridiculous when your producers 1. produce events in clean batches, and 2. could re-produce those events, in the exact same batches, if prompted.<p>Ideally, rather than having to basically have a whole second DBMS in the form of a Kafka cluster (stateful nodes with big HDDs, carefully backed up, needing repair when any die), what I would love is a Kafka-like &quot;event store with MQ semantics&quot;, which is in turn backed by an <i>object store</i>. Basically, like Datomic&#x27;s segment-storage architecture; or like WAL-E in its use-case as an intermediary for Postgres replication. Unlike either of those, though, MQ log-segments don&#x27;t need any additional metadata (e.g. checkpoints, indices), so they can just be chucked straight in object storage, and MQ nodes can base their view of the world entirely on what happens to be in the object-storage bucket.<p>Of course, this kind of system only makes sense if you have no need for durability of new messages (because e.g. the producer still has the data that went into them, and can resend them), only old messages. Which is kind of an unusual requirement, I suppose, so I don&#x27;t really expect anyone else to go building this sort of thing for me.