First test nonsignificant; second test significant. Do I need to include repeated testing as a factor in ANOVA? I am running a repeated measures ANOVA for my data set.
Our rats completed 8 training sessions in 2 conditions (within-subjects; 16 sessions total). Then they completed a test - the same test in each condition (2 tests). The test was not significant between conditions. So we ran three additional training sessions, thinking that perhaps 8 was just not enough for them to learn that the two conditions were different. So, now they have 11 total sessions in each condition (22 altogether). We ran the same test again in both conditions (so now they've had 4 iterations of the test). Alone, test 2 came back significantly different between the conditions. Every rat behaved in the same way, which is what we like to see. But, when we use the repeated test as a within-subjects factor in our two-way repeated measures ANOVA (condition x test), the effect disappears because it's not that different from the first test (meaning only a couple rats needed the extra three sessions). I'm under the impression that conservatively, we should use the test factor, but that it wouldn't be necessary; here's my thinking:
The test 1 came back significant. So theoretically, that's saying that there was no difference between conditions for our rats at that time. Since they had the test in both conditions, it's now counterbalanced, and test 2 can be analyzed separately. In animal research, we only run a few animals so we are lacking power. I think the estimation plots are useful here because the statistics on test 2 alone show that there is a difference in test scores for each condition and that 100% of the subjects acted in that way (this was not the case following test 1). Statistics are supposed to support trends we see in the data, so I think it is fine to leave out the repeated test factor in this case. What do you think?
We still ran a two way repeated measures anova (one for each test set 1 and 2), with the order of the tests as our second factor and the test results effect holds up for test 2.

What you're doing here sounds alot like p-hacking, and it's generally frowned upon. You're massively increasing the probability of a type I error.
What if the second test wasn't significant? Would you have carried on? How long would you have carried on?
There are ways to adjust your p-values with stopping rules, but to do this, you specify in advance when you will test and when you will stop.
This is why you do a power analysis in advance, and why you need a pre-specified analysis plan. Without that, we have no confidence that your results are replicable.
