Most Winning A/B Test Results Are Illusory [pdf]
I think the entire approach discussed in this pdf is flawed. (Edit: not saying PDF itself is flawed or wrong, just the hypothesis testing approach to A/B testing.)The right question to ask is: What is the difference between A and B, and what is our uncertainty on that estimate?The wrong question to ask is: Is A different/better than B, given some confidence threshold?The reason this is the wrong question is that it's unnecessarily binary. It is a non-linear transformation of information that undervalues confidence away from the arbitrary threshold and overvalues confidence right at the arbitrary threshold.A test with only 10 or 100 samples still gives you information. It gives you weak information, sure, but information nonetheless. If you approach the problem from a continuous perspective (asking how big the difference is), you can straightforwardly use the information. But if you approach the problem from a binary hypothesis-testing perspective (asking is there a difference), you'll be throwing away lots of weak information when it could be proving real (yet uncertain) value.Once you switch away from the binary hypothesis-testing framework, you no longer have to worry about silly issues like stopping too early or false positives or false negatives. You simply have a distribution of probabilities over possible effect sizes.