Understanding Statistical Power and Significance Testing
Here&#x27;s a good example of how p values are misleading:<p>Consider a bag containing two coins, one that&#x27;s fair and one that always shows heads. Pick one coin at random and flip it ten times. Suppose you see ten heads. Would you be comfortable saying that you have the biases coin? Most likely you would. Formally you&#x27;d compute the p value (in this case something like 1&#x2F;1024 or 2&#x2F;1024 depending on if you do a one sided or two sided test). You&#x27;d compare this with some cutoff, perhaps 0.05 and claim that you can reject the null hypothesis.<p>So far so good.<p>Now let&#x27;s perform that experiment again but this time instead of having one fair coin and one biased coin in the bag, suppose we have one biases coin and 10^1000 fair coins. Suppose again that you randomly pick a coin from the bag and flip it ten times. Again you see ten heads. Do you think you have the biases coin? Of course not! What&#x27;s the chance that you just happened to choose the single biased coin instead of one of the 10^1000 fair coins? Practically zero. In this case it&#x27;s much more likely that the ten heads were a statistical fluke from a fair coin. However, if you compute the p value you still get 1&#x2F;1024 (or 2&#x2F;1024 for a two sided test).<p>This example should be reasonably convincing that one needs access to a prior in order to correctly interpret p values. One way of looking at it is that you need to set the threshold appropriately based on the plausibility of the alternate hypothesis. Another way to phrase it is that we actually need to compute the posterior Bayesian probability of the null hypothesis being true.<p>There are tons of other confusing &#x2F; surprising examples where a naive p-value calculation will be misleading.<p>I wrote about these things here:
<a href="https:&#x2F;&#x2F;mindbowling.wordpress.com&#x2F;2016&#x2F;07&#x2F;19&#x2F;p-values&#x2F;" rel="nofollow">https:&#x2F;&#x2F;mindbowling.wordpress.com&#x2F;2016&#x2F;07&#x2F;19&#x2F;p-values&#x2F;</a>