Publishing with Apache Kafka at The New York Times
&gt;We need the log to retain all events forever, otherwise it is not possible to recreate a data store from scratch.<p><i>SIGH</i>. Cue the facepalm, head in hands, etc.<p>I&#x27;m not going to get into a big thing here. But if you find yourself saying &quot;I need to keep this thing forever no matter what&quot; and then you try to use something that even entertains the notion of automatic eviction&#x2F;deletion semantics as the system of record, <i>you&#x27;re doing it wrong</i>.<p>Not to burst the bubble of the techno-hipsters, but Kafka is &quot;durable&quot; relative to message brokers like RabbitMQ, not relative to a system <i>actually designed</i> to store decades of mission-critical data. Those systems are called &quot;RDBMS&quot;.<p>Elsewhere in the article he says that they have less than 100GB of data and that it&#x27;s mostly text. This is massive overarchitecture that isn&#x27;t even covering the basic flanks that it thinks it is, such as data permanence.<p>I would really like to read the article that discusses why Postgres or MySQL <i>couldn&#x27;t</i> have served this purpose equally well.