Storage Optimization, Software Architecture, AWS, Icebergs, Kafka.
the rows of interest that need to be deleted on that day, we export all of them to a Kafka queue using our homegrown solution for streaming data movement and processing — Data Mesh. We created a Data Mesh pipeline consisting of an Iceberg connector and Kafka sink. We run a couple of Garbage