Is it still necessary to correct for multiple comparisons/testing if the sample sizes are large? I was having a discussion with a colleague today regarding corrections for multiple testing. We're planning on running a large number of tests (probably in the hundreds as a rough estimate) and thus I brought up multiple testing and how we should go about it. Specifically the fact that various outcomes are related (not independent) and thus this doesn't meet the assumptions of a Bonferroni, Benjamini-Hochberg or whatever method of adjustment. Their suggestion was that we don't have to worry about it, because the sample size of each test is going to be big enough (we're looking at minimum of n=100 but frequently n=1000+). This is the first time I've ever heard of such an approach and thus was a bit surprised by this. From my research and trying to think it through myself, perhaps their logic was that as n gets larger, sampling error becomes less of an issue, therefore it counteracts any increase in false positives?
My question ultimately is, is my colleague correct in suggesting the large sample sizes are sufficient for handling multiple testing and if so, is my reasoning of it being related to the sampling error accurate?

A easy way to make sense of this is in terms of effect sizes, Type 1 errors, Type 2 errors, and Power.
Let's say that you're looking at correlations, and you have $N$ data points.

*

*Your effect size is the correlation coefficient, $r$.

*Your Type 1 error rate, $\alpha$, is the probability of concluding that there is an effect, $r \neq 0$, when there really is no effect, $r = 0$. We usually use $\alpha = .05$, which means we conclude there is an effect (reject the null hypothesis) when the p-value is $p< .05$. When you conduct multiple tests, you usually adjust for multiple comparisons to keep the Type 1 error rate at $0.05$.

*Your Type 2 error rate, $\beta$, is the probabiliy of failing to conclude that there is an effect when there really is one. Your Power is just the inverse of this, $1 - \beta$.

Now, what happens with different sample sizes?
Here's a table of critical values, from real-statistic.com:

Focus on the third column, where $\alpha = 0.05$.
The values in each row show, for each sample size
(these rows actually show degrees of freedom, which is just $N - 2$),
how strong the correlation $r$ needs to be in order to be significant,
$p < .05$:

*

*When $N = 12$ (df $= 10$), $r \geq 0.57$ is significant.

*When $N = 102$ (df $= 100$), $r \geq 0.19$ is significant.

*When $N = 1002$ (df $= 1000$), $r \geq 0.06$ is significant.

Crucially, the Type 1 error rate is the same in each case, $\alpha = 0.05$.
Increasing the sample size has instead made it possible to detect smaller effects, which reduces the Type 2 error rate/boosts power (you're less likely to miss a small but non-zero effect).
Finally, adjusting for multiple comparisons just means
using a more conservative threshold for how big $r$ needs to be
before you decide there's an effect. In other words, to reduce your Type 1 error rate, you have to increase your Type 2 error rate. Luckily, collecting lots of data means your Type 2 error rate is low, so you can better afford to do this.
So, putting this together:

*

*More data reduces Type 2 errors, but leaves Type 1 errors unchanged at $\alpha = 0.05$.

*Doing multiple comparisons increases Type 1 errors

*To counteract this, you adjust for multiple comparisons by making your decisions more conservative. This brings the Type 1 error rate back down, at the expense of increasing the Type 2 error rate.

