Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)
For what it's worth, I've been following this cross-Internet debate with more than a little professional interest.  Cards on the table: I have coded A/B testing software, I frequently code and/or administer it for clients (often in ways which are provably suboptimal), and I am a dirty loyalty-free scientist-cum-capitalist-pig who would stab A/B testing in the back in a second if I thought there were an easier way to extract more money for the same amount of work.<p>I strongly, strongly suggest that anyone attempting to look at this problem from the perspective of a site owner rather than a mathematical abstraction read and digest btilly's comment from earlier this week:<p><a href="http://news.ycombinator.com/item?id=4040616" rel="nofollow">http://news.ycombinator.com/item?id=4040616</a><p>The issues he lays out are <i>very real in the course practical use of site testing to actually make money</i>.  In particular, his #2 would scare the heck out of me, in a much deeper way than "A/B testing provably doesn't minimize regret" worries me in the other direction.  (Or e.g. other flaws with particular A/B testing implementations.  For example, repeatedly checking the results of your A/B test and deciding to end it when you see significance has been explained quite a few times as a bad idea with stats to match.  However, even if you check like a hyperactive squirrel, you're still winning, you're just winning less often than you think you are.  Take your B- in stats class but proceed to make motivational amounts of money for the business.)<p>The worst possible takeaway you, personally, the typical HN reader, could possibly have from this debate is "Oh, I guess I shouldn't A/B test then."  Pick A/B testing, bandit testing, whatever -- any option in the set, even with poor algorithms and/or the easiest errors I can think of, strictly dominate not testing at all.  (Actually testing today also is better than "testing... someday", which from my own experience and that of clients I know is something which is very easy to slip into even if you theoretically know you should be doing it.)