Why Multi-armed Bandit algorithms are superior to A/B testing (with Math)
For what it's worth, I've been following this cross-Internet debate with more than a little professional interest.  Cards on the table: I have coded A/B testing software, I frequently code and/or administer it for clients (often in ways which are provably suboptimal), and I am a dirty loyalty-free scientist-cum-capitalist-pig who would stab A/B testing in the back in a second if I thought there were an easier way to extract more money for the same amount of work.I strongly, strongly suggest that anyone attempting to look at this problem from the perspective of a site owner rather than a mathematical abstraction read and digest btilly's comment from earlier this week:http://news.ycombinator.com/item?id=4040616The issues he lays out are very real in the course practical use of site testing to actually make money.  In particular, his #2 would scare the heck out of me, in a much deeper way than "A/B testing provably doesn't minimize regret" worries me in the other direction.  (Or e.g. other flaws with particular A/B testing implementations.  For example, repeatedly checking the results of your A/B test and deciding to end it when you see significance has been explained quite a few times as a bad idea with stats to match.  However, even if you check like a hyperactive squirrel, you're still winning, you're just winning less often than you think you are.  Take your B- in stats class but proceed to make motivational amounts of money for the business.)The worst possible takeaway you, personally, the typical HN reader, could possibly have from this debate is "Oh, I guess I shouldn't A/B test then."  Pick A/B testing, bandit testing, whatever -- any option in the set, even with poor algorithms and/or the easiest errors I can think of, strictly dominate not testing at all.  (Actually testing today also is better than "testing... someday", which from my own experience and that of clients I know is something which is very easy to slip into even if you theoretically know you should be doing it.)