Most Winning A/B Test Results are Illusory [pdf]
At my first webdev internship, my only job was to report to the "Head of Analytics" (a young liberal arts guy). All I did all day was make the tweaks he told me to do. It was stuff like "make this button red, green or blue", or "try these three different phrasings".We got no more than 100 hits a day, with no more than 2-3 conversions a day, and he would run these tests for, like, 2 days.I hated it, and the website looked horrible because everything was competing with each other and just used whatever random color won.