Most Winning A/B Test Results Are Illusory [pdf]
I think the entire approach discussed in this pdf is flawed. (Edit: not saying PDF itself is flawed or wrong, just the hypothesis testing approach to A/B testing.)<p>The right question to ask is: What is the difference between A and B, and what is our uncertainty on that estimate?<p>The wrong question to ask is: Is A different/better than B, given some confidence threshold?<p>The reason this is the wrong question is that it's unnecessarily binary. It is a non-linear transformation of information that undervalues confidence away from the arbitrary threshold and overvalues confidence right at the arbitrary threshold.<p>A test with only 10 or 100 samples still gives you information. It gives you weak information, sure, but information nonetheless. If you approach the problem from a continuous perspective (asking <i>how</i> big the difference is), you can straightforwardly use the information. But if you approach the problem from a binary hypothesis-testing perspective (asking <i>is</i> there a difference), you'll be throwing away lots of weak information when it could be proving real (yet uncertain) value.<p>Once you switch away from the binary hypothesis-testing framework, you no longer have to worry about silly issues like stopping too early or false positives or false negatives. You simply have a distribution of probabilities over possible effect sizes.