Most Winning A/B Test Results are Illusory [pdf]
At my first webdev internship, my only job was to report to the "Head of Analytics" (a young liberal arts guy). All I did all day was make the tweaks he told me to do. It was stuff like "make this button red, green or blue", or "try these three different phrasings".<p>We got no more than 100 hits a day, with no more than 2-3 conversions a day, and he would run these tests for, like, 2 days.<p>I hated it, and the website looked horrible because everything was competing with each other and just used whatever random color won.