On Sharding
> Load-sensitivity is one “smart” approach. The idea is that you keep track of the load on each shard, and selectively route traffic to the lightly-loaded ones and away from the busy ones. Simplest thing is, if you have some sort of load metric, always pick the shard with the lowest value.<p>Gotta be super careful with this one.  We did this at reddit and it bit us bad.  The problem was as soon as the load on a machine went down it got pounded with new requests and the load shot up, but it takes a few seconds for the load number to react to all the new requests.  So we saw really bad see-saw affect.<p>We had to add extra logic to mark how long a machine had beed at a certain load and also randomly send requests to slightly more loaded machines to keep things even.<p>The moral of the story here is make sure you pick a metric that reacts to the change in request rate as quickly as your request rate changes!