Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale
The most literal conclusion to draw from this story is that MRU connection pools shouldn’t be used for connections that traverse aggregated links.Not just connections which traverse aggregated links.  If you're load-balancing between multiple database replicas -- or between several S3 endpoints -- this sort of MRU connection pool will cause metastable load imbalances on the targets.What I don't understand is why Facebook didn't simply fix their MRU pool: Switching from "most recent response received" to "most recent request sent" (out of the links which don't have a request already in progress, of course) would have flipped the effect from preferring overloaded links to avoiding them.