Over the last week or so I have been asked a couple of
statistical questions where the root cause of the problems has
been the unthinking application of p-values. In both cases, the
solution was to use confidence intervals instead; i.e. to look
at the size of the effect, not whether it was different from
zero. I decided it was time to take the battle to the host of
p-values, armed only with a t-test. It is enough. What is so bad
about p-values? Let’s start with a question: what does a p-value
tell you? If you answered “the probability of getting the
observed statistic, or something more extreme, if the null
hypothesis was true”, then you’re right but you’re probably only
parroting the textbook. If you said something about it giving
the strength of evidence that a hypothesis is correct, then
you’re wrong, but in good company: that is how most people use
them in practice. Time to spring an example on you. Let’s
suppose we are interested in whether rocket scientists are more
intelligent than brain surgeons. So,