Why does $\mu > 0$ (or even $\mu > \epsilon$) "seem easier” to substantiate than $\mu \neq 0$? Consider a random variable $X$ following a normal distribution $N(\mu,\sigma^2)$. Suppose that we have drawn iid samples of $X$, obtaining a data set with a sample mean $\bar{x}>0$.
We want to test whether the data supports $\mu >0$ with a significance level $\alpha \in (0,1)$. Setting
\begin{equation*}
H_0: \mu \le 0, 
\quad H_1: \mu > 0, 
\end{equation*}
we examine the $p$-value
\begin{equation*}
p \;=\; \Pr(\overline{X} \;\ge\; \bar{x} \mathrel{|} H_0) \;=\; \Pr(\overline{X} \ge \bar{x}\mathrel|{\mu \le 0}) \;\le\; \Pr(\overline{X} \ge \bar{x} \mathrel{|} \mu = 0).
\end{equation*}
[N.B.: As kindly pointed out by @dimitriy, in practice we often set the null hypothesis simply to $\mu=0$, for which the $p$-value would be directly the RHS of the above inequality.]
As in textbooks, the data provides a sufficient evidence to substantiate $\mu > 0$ if $p \le \alpha$.
Now, we change our mind and test whether the data supports $\mu \neq 0$, with
\begin{equation*}
\hat{H}_0: \mu = 0, 
\quad \hat{H}_1: \mu \neq 0,
\end{equation*}
and the $p$-value being
\begin{equation*}
\hat{p} \;=\; \Pr(|\overline{X}| \ge \bar{x} \mathrel{|} \hat{H}_0) \;=\; \Pr(|\overline{X}| \ge \bar{x} \mathrel|{\mu = 0}).
\end{equation*}
Again, textbooks tell us that the data provides sufficient evidence to substantiate $\mu \neq 0$ if $\hat{p} \le \alpha$, with $\alpha$ being the aforementioned significance level.
Here comes the part that confuses me. Since
$$\hat{p} \ge 2p,$$
it turns out "easier" to substantiate $\mu >0$ than $\mu \neq 0$, in the sense that the data may strongly support $\mu >0$, but does not constitute sufficient evidence for $\mu \neq 0$; equivalently, the data may strongly suggest rejecting $\mu \le 0$ without being able to reject $\mu = 0$. For instance, imagine that $p \le 0.6\%$, $\hat{p} = 1.2\%$, and $\alpha = 1\%$.
[Update: We can strengthen the above observation a bit: there exists an $\epsilon >0$ that it “seems easier” to substantiate $\mu > \epsilon$ than doing the same for $\mu \neq 0$.]
So what is the logic? What have I overlooked or misunderstood? I believe the calculation is trivial to everyone here, but there seems to be something to say about the interpretation. What is it?
In my textbook, hypothesis testing is compared to a court trial, with the null hypothesis being the suspect is innocent. Following this comparison, a probably quite poor metaphor for the paradox elaborated above is that we may have strong evidence to support that "A killed B by stabbing a dagger into B's chest", but not enough justification to substantiate that "A killed B". No, I must have made some silly mistake.
Thank you very much.
Background: Being not a statistician by any means, I am asked to teach a course that involves a part of statistics. The question occurs to me when I am preparing my notes.


Hypothesis testing: why $\mu > 0$ (or even $\mu > \epsilon$) "seems easier” to substantiate than $\mu \neq 0$?

It seems easier because the one-sided t-test and two-sided t-test have different sensitivity for different values.

*

*The two-sided t-test has sensitivity split for both positive and negative values.

*The one-sided t-test has sensitivity for only the positive or only the negative values, and because of that will be 'easier' in substantiating a result (but only for a single side)

Below is a graph of the sensitivity or power for the two tests as a function of the true mean of the distribution. (Power is the probability that an observation is a positive result).

You can see that the one-sided test is not everywhere 'easier' than the two-sided test. It is only 'easier' for the positive values.

Also note that the tests have an equal $5\%$ probability/frequency of a positive result when in reality the effect is negative (when the true mean is equal to zero). So if the true mean is equal to zero then the two hypothesis tests are equally 'easy' in making a false claim of substantiating an alternative hypothesis (like $\mu \neq 0$ or $\mu > 0$).
But they are still different. They will make these (false) claims for different observations and for a specific observation they are not equally 'easy'.
This can occur because there is no unique way to compute p-values and associated hypothesis tests. The p-values based on different methods can be different for a particular observation but on average (for all possible observations) they will be equal.
Below you see a simulation of $10 000$ samples of size $n=5$ drawn from a standard normal distribution. We plot the observed (unbiased) sample standard deviation $s$ and the observed sample mean. Along with it we plot the rejected sampled based on a two-sided t-test and a one-sided t-test.

*

*The amount of rejected samples is in both cases the same, namely $5\%$.

*But, the rejected samples are different for the cases. The one-sided t-test is more sensitive to values on one side. The two-sided t-test splits the sensitivity to two sides.

So again, that is why the one-sided test may seem more 'easy' but that is because of the choice to place the sensitivity in a different region. It is only 'easier' for that region.

So it may occur that you reject $H_0: \mu \leq 0$ with an alternative hypothesis $H_a: \mu > 0$, but you can't reject with the same data $H_0: \mu = 0$ with an alternative hypothesis $H_a: \neq 0$.
When such situation occurs then it is seemingly a paradox. But,

*

*It is not purely the data that rejected the $H_0$.

*It is also your choice and it is also the alternative hypothesis that 'helps' rejecting the null hypothesis.

The same hypothesis, with the same data, can be or not be rejected, depending on your arbitrary rejection criteria.
Other examples (from this website) were different tests reject a hypothesis for different observations are:

*

*ANOVA with F-test vs Tukey's range test. The ANOVA test and Tukey's procedure test the same null hypothesis 'equality of means' but have different p-values for different observations because they relate to different statistics and have sensitivity in different regions. One looks at the largest difference between samples, the other at the variance.


*Mann-Whitney U test versus t-test can be used to test equality of two means. They have different p-values because the one uses the t-statistic based on the ratio of the difference in the means and the standard deviation, the other computes a statistic based on how often a value in the one sample is larger than a sample in the other sample.


*Fisher exact test with pooled versus non-pooled data. In that question the Fisher exact test had a surprising behaviour in the power being different when the data was split into groups. But, effectively this was due to using different regions where the tests are sensitive.
