Low p-value, but many trials I am not well-versed in statistics, so please bear with me. I will try to explain things to the best of my knowledge.
I have some hardware that produces a signal, this signal has an offset and I want to subtract this baseline from my signal. This is the procedure I use to determine the baseline:
I take many noise samples (with long time between them to avoid any correlation) and from these I obtain the mean (my baseline) and the standard error on the mean (sample standard deviation / sqrt(samples)).
This looks good, but I now want to determine when I need to calibrate my baseline again, so I repeat the same procedure and obtain a new_mean, new_std_error, new_number_of_samples. I perform a two-tailed Welch's unequal variances t-test on these and if the p-value is low enough (say 0.05) I determine that my baseline has changed and I need a new calibration.
What is the problem?
My device has 256 channels, and if I perform this procedure on all channels I will most certainly obtain a low p-value for a couple of channels by pure luck. I don't think that this means these channels need a new baseline.
Other device has 20 thousand channels, even more likely to get a low p-value for some channels. Do you have any suggestion on how to deal with this? How could I determine if there is any actual reason to update the baseline of any given channel?
Thanks

From a purely statistical perspective, this is exactly what multiple testing correction is for. The first "design" question for you is what exactly you would like to control:

*

*Either you want to keep the probability that there is any false positive (i.e. spuriously significant test) below a certain threshold $\alpha_1$ - this leads to the so-called Bonferroni correction method, where you essentially multiply your raw p-values with the number of tests (in fact, there exists a number of variations on this, but all of them share this goal and principle)

*Alternatively, you could say that of your "hits" of low p-values, you'd like, in expectation, at most a certain fraction $\alpha_2$ to be false positives, i.e. randomly spuriously significant tests, so that on average a proportion of $(1-\alpha_2)$ of your significant tests results from a real signal. This leads you to the False-Discovery-Rate family of methods, which prescribe a somewhat more complex procedure to correct your raw p-values

The former is evidently much more conservative than the latter, but there is no "objectively correct" choice - rather it really depends on the tradeoff between false positive and false negative rate. From your description, it seems that the FDR correction could be more suitable to your problem, but the decision ultimately rests with you.
