Kafka Is Not a Database
As recently as last year, I worked for a company where the Chief Architect, in his infinite wisdom, had decided that a database was a silly legacy thing. The future looked like Kafka streams, with each service being a function against Kafka streams, and data retention set to infinite.<p>Predictably, this setup ran into an interesting assortment of issues. There were no real transactions, no ensured consistency, and no referential integrity. There was also no authentication or authorization, because a default-configured deployment of Kafka from Confluent happily neglects such trivial details.<p>To say this was a vast mess would be to put it lightly. It was a nightmare to code against once you left the fantasy world of functional programming nirvana and encountered real requirements. It meant pushing a whole series of concerns that isolation addresses into application code... or not addressing them at all. Teams routinely relied on one another&#x27;s internal kafka streams. It was a GDPR nightmare.<p>Kafka Connect was deployed to bridge between Kafka and some real databases. This was its own mess.<p>Kafka, I have learned, is a very powerful tool. And like all shiny new tools, deeply prone to misuse.