Whom the Gods Would Destroy, They First Give Real-time Analytics
Gah, yet another article that links to Evan Miller's article on how to not run an A/B test.  I really need to finish writing my article that explains why it is wrong, and how you can do better without such artificial restrictions.<p>His math is right, but the logic misses a basic fact.  In A/B testing nobody cares if you draw a conclusion when there is really no difference, because that is a bad decision that costs no money.  What people properly should care about is drawing the wrong conclusion when there is a real difference.  But if there is a significant difference, only for small samples sizes is there a realistic chance of drawing a wrong conclusion, and after that the only question is whether the bias has been strong enough to make the statistical conclusion right.<p>He also is using 95% confidence as a cut-off.  Don't do that.  You don't need much more data to massively increase the confidence level, and so if the cost of collecting it is not prohibitive you absolutely should go ahead and do that.  Particularly if you're tracking multiple statistics.  If you test regularly those 5% chances of error add up fast.