7 Years Of YouTube Scalability Lessons In 30 Minutes 
Most of this is relatively straightforward and unsurprising.  But the one part that grabbed me is about "jittering".  They insert random delays into timed events (the example given is cache expiration) to prevent a thundering herd problem when all the parts of the distributed system see the event at the same time (and for popular content, presumably repopulate the cache from the backend simulteously).<p>This is simple enough when described, but is not a technique I've seen applied much in practice or discussed in the community.  I'm wondering if it's something that gets reinvented for all the projects that need it or if it's secret sauce known only in youtube.  Regardless, I thought it was pretty insightful.