Designing Data-Intensive Applications
In "The Future of Data Systems", the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work like de-duping, committing the data to a RDBMS, invalidating caches, updating search indexes, etc.  The application might read state directly from log updates, or have a system to sync w/ some sort of atomic state (e.g. RethingDB changefeeds).The architecture seems to solve two big problems* Scaling RDBMS (there are solutions like Cloud Spanner but they rely heavily on Google's proprietary network for low latency and are expensive as balls)* Keeping downstream systems in sync. A lot of companies have gone with a "Tail the RDBMS" of some kind, e.g. writing the MySQL binlog to a Kafka queue and having downstream consumers read from that, but this seems like a more elegant solution.Are there any examples or experiences of people working with systems like this? What are some downsides, challenges, and actual benefits?