What does it mean for tests to be independent? When reading about cumulation if type-1 Error, the sentence "for independent statistical tests" occures alot, now I was wondering what this is actually means.
Since tests are also random variables on the sample space, the definition of independence can be applied, mathematically. Now lets look at two tests: $t_1,t_2$, the definition of independence could be:
$P(t_1 \in A, t_2 \in B) = P(t_1 \in A) \cap P(t_2 \in B)$ for alle sets $A,B$ in $\mathcal{B}(\mathbb{R})$. Since tests can only assume discrete values, the sets $A,B$ can in reality only be discrete sets consiting of $0,1$ and sometimes one more values.
Now so far so good, but recently I read a paper in which the following sitation occured:
A test rejected the $0$ hypothesis, then the test was repeated on a different sample. The author then stated, that the probability for a type-1 Error in this "repeat test setup" was $\alpha^2$, because of the independence of the tests. I understand why this is true if I acknowledge the "independent" part, but this is the part I have my problems with.
To complete my post, here is the paper the statement is from, it is a download link. The statement is made on page 2 in the last paragraph.

This is really talking about conditional independence, with conditioning on the hypothesis.
So if the null hypothesis is correct, then the probability the test is significant should be $\alpha$.  If you then collect new data and test the new data, the probability the second test is significant should also be $\alpha$, and the conditional independence means the probability of both tests being significant is $\alpha^2$.
Similarly, if the alternative point hypothesis is correct, and the test has power $1-\beta$, i.e. that is the probability of a significant result, then the probability of two significant results is $(1-\beta)^2$ for the same reason.
But if you do not know which hypothesis is correct then you do not get this independence. If in a quasi-Bayesian approach you think there is a probability $p$ that the null hypothesis is correct and a $1-p$ probability that the alternative point hypothesis is correct, then the probability the first test is significant is $p\alpha + (1-p)(1-\beta)$, and that both are significant is $p\alpha^2 + (1-p)(1-\beta)^2$ which is not the square of the previous expression when $0 < p<1$. This is the concern you express in your question.
You also need to consider what you would do if the first test was significant and the second not, which happens with probability $\alpha(1-\alpha)$ or $\beta(1-\beta)$ depending on which hypothesis is correct.
