Low p-value, but many trials I am not well-versed in statistics, so please bear with me. I will try to explain things to the best of my knowledge.
I have some hardware that produces a signal, this signal has an offset and I want to subtract this baseline from my signal. This is the procedure I use to determine the baseline:
I take many noise samples (with long time between them to avoid any correlation) and from these I obtain the mean (my baseline) and the standard error on the mean (sample standard deviation / sqrt(samples)).
This looks good, but I now want to determine when I need to calibrate my baseline again, so I repeat the same procedure and obtain a new_mean, new_std_error, new_number_of_samples. I perform a two-tailed Welch's unequal variances t-test on these and if the p-value is low enough (say 0.05) I determine that my baseline has changed and I need a new calibration.
What is the problem?
My device has 256 channels, and if I perform this procedure on all channels I will most certainly obtain a low p-value for a couple of channels by pure luck. I don't think that this means these channels need a new baseline.
Other device has 20 thousand channels, even more likely to get a low p-value for some channels. Do you have any suggestion on how to deal with this? How could I determine if there is any actual reason to update the baseline of any given channel?
Thanks

When deciding whether to adjust the baseline you need to consider more than just the long run rate of adjusting it when that channel does not actually need it.
The p-value is an index of the strength of evidence in the data against the null hypothesis according to the statistical model. In the hypothesis testing framework you make a decision on the basis of the relationship between that p-value and a pre-determined fixed threshold. That threshold should be set to a level that is chosen to optimise not only the long run rate of false positives (i.e. channel adjustments that are made but not needed), but also the power to detect false nulls (i.e. detect occasions where a channel needs adjustment) and the necessary sample size. The considerations of threshold and sample size involve thinking about the consequences of false positive and false negative errors. DO NOT use p=0.05 as a threshold and think that you are 'doing' statistics!
What are the consequences of adjusting the baseline of a channel that does not actually need adjustment? If they are trivial then adjust when there is modest evidence against the null hypothesis that the baseline is correct. What are the consequences of failing to adjust an out of baseline channel?
I would suggest that you consider a procedure where you monitor the channels and whenever a channel's p-value (or other index of out of balance condition) is low enough you sample again and adjust if a second p-value is also low. The chances of getting a spuriously low p-value two times in a row is very low, but if the channel needs adjustment then it should regularly give low p-values. That way you will hugely reduce the rate of false positive errors without also greatly reducing the ability to detect the need to adjust. (This assumes that a fresh sample of values has minimal cost.)
On a non-statistical level, why not use an AC-coupling approach (capacitative coupling) for the hardware? That is often used when there can be baseline drift and it deals with the problem very gracefully. It can also be done digitally. See here: https://en.wikipedia.org/wiki/Capacitive_coupling
