p-value and "by chance alone" vs. the misinterpretations I know this topic was explained many times, but I need help with just a concrete wording.
p-value is the probability of obtaining at least as extreme data (or test statistic for convenience), calculated UNDER the true null hypothesis.
That is - "no effect", "by chance" (nothing else by chance could create the effect) was assumed, and then we looked at the collected data, to see, if they "support" or "match" that claim or not. In other words, it's Probability(data | true null hypothesis).
Let's play with words about the p-value:

*

*Probability, that as extreme or more extreme data were collected provided that H0 was actually true.


*Probability, that as extreme or more extreme data were collected provided that nothing else but chance "operated" or "acted"


*Probability, that the observed data can be explained by chance alone (we assume that nothing happened, but such data arrived)


*The result is explainable by chance alone.
At which point it becomes incorrect? To me, all sound exactly the same. I don't feel the subtle differences. By chance = nothing acts = true H0.
At the same time, the article: Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations says:

#2 The P value for the null hypothesis is the probability that chance
alone produced the observed association; for example, if the P value
for the null hypothesis is 0.08, there is an 8 % probability that
chance alone produced the association.
No! This is a common variation of the first fallacy and it is just as false. To say that chance alone
produced the observed association is logically equivalent to asserting
that every assumption used to compute the P value is correct,
including the null hypothesis. Thus to claim that the null P value is
the probability that chance alone produced the observed association is
completely backwards: The P value is a probability computed assuming
chance was operating alone. The absurdity of the common backwards
interpretation might be appreciated by pondering how the P value,
which is a probability deduced from a set of assumptions (the
statistical model), can possibly refer to the probability of those
assumptions.

OK, so saying, that p-value is probability of obtaining the data "by chance" is NOT the same, as saying that "such or more extreme data were obtained assuming that only chance operated"?
At the same time, a book that I am just reading, "Understanding Regression Analysis: A Conditional Distribution Approach" by Westfall & Ariar, which is very strict about the misunderstanding of p-values, claims, that p answers the question: "Is this result explainable by chance alone?"
Google Books shows the following, but I can hardly understand how these subtle wordings are correct, if the "by chance alone" is indicated as WRONG in the cited article as misinterpretation?

*

*
and from page 84:

When the only reason for a difference between statistical estimates is
chance alone, and  not  any  systematic  effect,  then  that
difference  is  said  to  be  explained  by  chance  alone.

and 85

Definition of “Explainable by chance alone”
When a difference between
statistical estimates is within a typical range of differences that
are explained by chance alone, then that difference is said to be
explainable by chance alone

Another book "AP Statistics Premium: With 9 Practice Tests" by Sternstein says:

All the authors are PhD in statistics, all say similar (to me) things. Who is then correct!?
I know, that p-value is a conditional probability about the DATA, not the hypothesis. We assume true H0 and check the data we got under this assumptions.
Isn't then THE SAME as saying that "how probable was to get such data under the true H0" = "how probable was to get such data only by chance" = "were the data explainable by chance alone"?
Please help, where's this subtle moment I miss?

Let's say that we did a t-test of any continuous real world measure was zero. No continuous measure is really zero to the millionth digit of 0.0000000... but the null hypothesis states it is zero no matter how many digits are in regard.
So we know beforehand, that the (two-tailed) null hypothesis is wrong! Bear with me, we test against the null of which we know, that is wrong. Surprisingly, there is no logic flaw in that.
When you bare that in mind, you can hopefully see how

there is an 8 % probability that chance alone produced the association.

Is wrong, because we know from before, that the probability of that is zero.
On the other hand

saying that "such or more extreme data were obtained assuming that only chance operated"?

Does not give the null any actual probability but just makes a counterfactual assumption.
I hope this point of view makes a difference (not necessarily all differences) visible.
