Technology, Engineering, Kafka, Apache Kafka, Software Development.
To reduce the number of calls to Kafka brokers, the system maintains an internal cache of lags rather than calling Kafka brokers before publishing each message. Using lag data, a customized algorithm is designed to publish less traffic to partitions experiencing high lag and more to those with low