Designing Data-Intensive Applications
In &quot;The Future of Data Systems&quot;, the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work like de-duping, committing the data to a RDBMS, invalidating caches, updating search indexes, etc.  The application might read state directly from log updates, or have a system to sync w&#x2F; some sort of atomic state (e.g. RethingDB changefeeds).<p>The architecture seems to solve two big problems<p>* Scaling RDBMS (there are solutions like Cloud Spanner but they rely heavily on Google&#x27;s proprietary network for low latency and are expensive as balls)<p>* Keeping downstream systems in sync. A lot of companies have gone with a &quot;Tail the RDBMS&quot; of some kind, e.g. writing the MySQL binlog to a Kafka queue and having downstream consumers read from that, but this seems like a more elegant solution.<p>Are there any examples or experiences of people working with systems like this? What are some downsides, challenges, and actual benefits?