Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values
I agree as well!<p>Here is what probability theory teaches us.  The proper role of data is to adjust our prior beliefs about probabilities to posterior beliefs through Bayes&#x27; theorem.  The challenge is how to best communicate this result to people who may have had a wide range of prior beliefs.<p>p-values capture a degree of surprise in the result.  Naively, a surprising result should catch our attention and cause us to rethink things.  This is not a valid statistical procedure, but it IS how we naively think.  And the substitution of a complex question for a simpler one is exactly how our brains are set up to handle complex questions about our environment.  (I&#x27;m currently working through <i>Thinking Fast and Slow</i> which has a lot to say about this.)<p>Simple Bayesian approaches take the opposite approach.  You generally start with some relatively naive prior, and then treat the posterior as being the conclusion.  Which is not very realistic if the real prior was something quite different.<p>Both approaches have a fundamental mistake.  The mistake is that we are taking a data set and asking what it TELLS us about the world.  When in probability theory the real role of data is how to UPDATE our views about the world.<p>This is why I have come to believe that for simple A&#x2F;B testing, thinking about p-values is a mistake.  The only three pieces of information that you need is how much data you are willing to collect, how much you have collected, and how big the performance difference is.  Stop either when you have hit the maximum amount of data you&#x27;re willing to throw at the test, or when the difference exceeds the square root of that maximum amount.  This is about as good as any simple rule can do.<p>If you try to be clever with p-values you will generally wind up putting yourself at saving yourself some effort in return for a small risk per test of making very bad mistakes.  Accepting small risk per test over many tests for a long time puts you at high odds of eventually making a catastrophic mistake.  This is a very bad tradeoff.<p>I&#x27;ve personally seen a bad A&#x2F;B test with a low p-value rolled out that produced a 15% loss in business for a company whose revenues were in the tens of millions annually.  It.  Was.  Not.  Pretty.  (The problem was eventually found and fixed..a year later and after considerable turnover among the executive team.)