Independence-violation issue in pairwise measurements: Which test? Here is another problem in which independence might be an issue (however, the question might be very basic).
Suppose, I have a rowing team and I have two boats (B1 and B2, for brevity). I have strong reason to believe that the team performs better with B1 than B2, so I want to compute a significance test.
This is would be the design:
First, there is a (balanced) measurement of the team's performance over the course of a two days, i.e. either B1 on day 1 and B2 on day 2, or B1 on day 2 and B2 on day 1 (there is sufficient time for the team to recover between the two measurements). Then, there is a period of training. Subsequently, the measurements are repeated.
With this procedure, I obtain two "time series" with race times; one for B1 and one for B2, over time. Is it permissible to compute a simple dependent measures t-test?
Normally, a repeated measures ANOVA would be the method of choice, but since for each time point, there is only one data point, it is not feasible. However, the pairwise t-test assumes samples to be statistically independent which they are not, given that they originate from the same athletes and each measurement captures a cumulative amount of training. Conversely, it could be argued the measurement pairs are somewhat independent, because e.g. fatigue from one measurement does not influence the next one, etc.
Alternatively, it might be possible/permissible to compute a permutation test, i.e. in which I keep the time points constant, however, I switch 50% of the (B1 and B2) labels randomly. As a measure, I take something simple like the sum of the element-wise difference between my B1- and B2-curve.
Here, independence might not be such a great problem, as I only take into account pairwise differences, and I carry whatever dependence they may have over the time of measurements into every random sample I generate.
Therefore, my p-value should be somewhat valid, shouldn't it?
If so: How can I compute a (reliable) effect size, based on this test?
Thank you for taking the time to read.
Best wishes

Here is one possible analysis.
As always, assumptions are made. These assumptions may or may not be reasonable in your actual application. (The boat rowing scenario is fun if fanciful.)
Let performance at time $t$ be the sum of long-term performance $\mu(t)$ and boat-advantage $\alpha_{B_i}(t)$:
$$
\begin{aligned}
\operatorname{E}(Y_t) = \mu(t) + \alpha_{B_i}(t)
\end{aligned}
$$
where time $t$ is "compressed" by 2 days, so that we can measure performance with both boats at the "same time". This time compression assumes that racing over the course of two days is not affected by the order of the boats, there is enough recovery time in between and the race conditions are the same.
I don't specify a functional form $\mu(t), \alpha_{B_1}(t)$ and $\alpha_{B_2}(t)$, though you imply that all three functions are monotonically increasing. The gist is that the effect of training more over time and the boat advantage are isolated from each other.
This model complies with your description of the situation (taken from a comment):

In the beginning, performances do not greatly differ and the difference emerges over practice (and with practice, the team becomes better with both B1 and B2, only more pronouncedly with B1)

Next we consider the difference in performance with the two boats:
$$
\begin{aligned}
\Delta_t = \alpha_{B_2}(t) - \alpha_{B_1}(t)
\end{aligned}
$$
The long-term performance conveniently cancels out and it remains to test the one-sided hypothesis: $H_0: \Delta \leq0$ against the alternative hypothesis $H_1: \Delta > 0$. Here we make the further assumption that enough time elapses between (paired) races, so that the observed differences are independent (but not identically distributed since boat advantage increases monotonically for both boats).
You can use the sign test (not very powerful) or the Wilcoxon signed rank test. We made plenty of assumptions already, so let's not make a Normality assumption in order to use the t-test.
