Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values
If they held the same meeting 20 times, would they reach the same conclusion in 19 of those meetings?<p>On a more serious note, I think that the use of the word &quot;significant&quot; to mean &quot;the effect is reasonably likely to exist by some standard&quot; should be abolished.<p>Webster&#x27;s 1913 dictionary says:<p>&gt; Deserving to be considered; important; momentous; as, a significant event.<p>Statisticians don&#x27;t use &quot;significant&quot; to mean important at all -- they use it to mean &quot;I could detect it&quot;.  This is bad when someone publishes a paper saying &quot;I found that some drug significantly reduces such-and-such&quot; -- this could just mean that they did a HUGE study and found that the drug reliably had some completely unimportant effect.  It&#x27;s much worse when it&#x27;s negated, though.  Think about all the headlines that say that some treatment &quot;did not have a significant effect&quot;.  This is basically meaningless.  I could do a study finding that exercise has no significant effect on fitness, for example, by making the study small enough.<p>A good friend of mine suggested that statisticians replace &quot;significant&quot; with &quot;discernible&quot;.  So next time someone does a small study, they might find that &quot;eating fat had no discernible effect on weight gain&quot;, and perhaps readers would then ask the obvious question, which is &quot;how hard did you look?&quot;.<p>This would also help people doing very good research make less wishy-washy conclusions.  For example, suppose that &quot;vaccines have no discernible effect on autism rates&quot;.  This is probably true in a number of studies, but <i>it&#x27;s the wrong analysis</i>.  If researchers who did these studies had to state the conclusions in a silly manner like that, maybe they&#x27;d find a more useful analysis to do.<p>Hint: doing big studies just so you can fail to find an effect is nonsensical.  Instead, do big studies so you can put a tight upper bound on the effect.  Don&#x27;t tell me that vaccines don&#x27;t have a significant (or discernible) effect on autism.  Tell me that, with 99.9% confidence, you have ruled out the possibility that vaccines have caused more than ten autism cases in the entire history of vaccines, and that, most likely, they&#x27;ve caused no cases whatsoever (or whatever the right numbers are).<p>Edit: fixed an insignificant typo.