Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale
<i>The most literal conclusion to draw from this story is that MRU connection pools shouldn’t be used for connections that traverse aggregated links.</i><p>Not just connections which traverse aggregated links.  If you're load-balancing between multiple database replicas -- or between several S3 endpoints -- this sort of MRU connection pool will cause metastable load imbalances on the targets.<p>What I don't understand is why Facebook didn't simply fix their MRU pool: Switching from "most recent <i>response received</i>" to "most recent <i>request sent</i>" (out of the links which don't have a request already in progress, of course) would have flipped the effect from preferring overloaded links to avoiding them.