Apache Kafka Beyond the Basics: Windowing
Queues like Kafka can be great for a certain class of solution, but I see them over used.<p>The justification is usually a sort of handwave about scaling, but I tend to be skeptical of that argument.<p>I remember in the early 2000&#x27;s writing a system that handled all of the consumer equipment orders and returns for Verizon, pick pack and ship and advanced billing integration with the ILEC phone bill, credit systems, IVR, etc... On a PII class server running SQL server stored procedures and a .net UI.<p>In 2009, I was processing a TB per month of 837 claims on even worse hardware.<p>Not that long ago, I saw a post on HN where a Casio calculator was handling the front page traffic.<p>My point is that even modest hardware has the capability to handle massive loads when the software is designed properly.<p>Kafka and it&#x27;s ilk are great for certain use cases, but it&#x27;s normally a mistake to start out with it unless you&#x27;re really sure you know what you&#x27;re doing.<p>CQRS and data-bus&#x2F;event sourced systems can be tricky to get right, and you lose a lot when you go to async processing, especially when submitting transactions from a user-facing app.<p>Inevitably you have to create some sort of polling or server push mechanism to update the UI, and the number of edge cases that can bite you are vast.