Does size of p-value matter beyond knowing if it is greater/less than a predetermined significance level? In my understanding, hypothesis tests are equivalent to confidence intervals. Because of this, I initially believed $p$-values should only be reported in a binary sense i.e. is $p<\alpha$
or $p>\alpha$  so you can either reject or fail to reject the null respectively.
But what happens when you do your test at the 5% level and is not significant at 5% level but at the 10% level? That means we are 90% confident that the true $\mu$ lies outside the 90% confidence interval, which seems quite high.
Is it not valid to reject at the 10% level in that case? Maybe I was stringent with the 5% significance.
Also since these cases only occur when $p$-values are between 2 significance levels (at least in the case of symmetric tests), doesn't that mean the size of $p$-values actually matters?

This question or questions very like it come up a lot. Your issue is that there are two distinct and incompatible ways of testing hypotheses using p-values, and many textbooks contain a confusing mix of the two. There is already an excellent explanation of the two approaches in this answer, but I'll summarise it (badly) here:

*

*In Fisher's approach, you calculate your p-value, but you don't compare it to .05, .1, or any other threshold. Instead, you just report it as it is, and a smaller p-value means stronger evidence against the null hypothesis.


*In the Neyman-Pearson approach, you have a clear set of decision rules (e.g. rejecting the null hypothesis if $p < \alpha$), which, if followed correctly, will limit the rate at which you make type 1 and type 2 errors.
So, you can either take Fisher's approach, and worry about the actual p-value, or Neyman and Pearson's, and worry about whether or not $p < \alpha$, but you can't do both.
