Low p-value, but many trials I am not well-versed in statistics, so please bear with me. I will try to explain things to the best of my knowledge.
I have some hardware that produces a signal, this signal has an offset and I want to subtract this baseline from my signal. This is the procedure I use to determine the baseline:
I take many noise samples (with long time between them to avoid any correlation) and from these I obtain the mean (my baseline) and the standard error on the mean (sample standard deviation / sqrt(samples)).
This looks good, but I now want to determine when I need to calibrate my baseline again, so I repeat the same procedure and obtain a new_mean, new_std_error, new_number_of_samples. I perform a two-tailed Welch's unequal variances t-test on these and if the p-value is low enough (say 0.05) I determine that my baseline has changed and I need a new calibration.
What is the problem?
My device has 256 channels, and if I perform this procedure on all channels I will most certainly obtain a low p-value for a couple of channels by pure luck. I don't think that this means these channels need a new baseline.
Other device has 20 thousand channels, even more likely to get a low p-value for some channels. Do you have any suggestion on how to deal with this? How could I determine if there is any actual reason to update the baseline of any given channel?
Thanks

This is closely related to the concept of change detection.
For example the CUSUM algorithm may be suitable. Basic implementation is:

*

*Cumulatively sum (sample - current_baseline) as data arrives.

*For every sample subtract some fraction of the standard deviation from the cumulative sum.

*If sum goes below 0, clamp it to 0. If sum goes above N times standard deviation, consider a change to have occurred.

Compared to taking discrete samples and computing statistics on them, a continuous process will react faster to a consistent change, such as new baseline value. The cumulative structure also means that small changes in baseline will still be eventually be detected, even if they don't immediately cross the threshold.
If you prefer the statistics structure, you can use a similar cumulative approach: expand the new sample count as long as the baseline is in effect. The longer this goes on, the smaller p will become and eventually it will cross even a very low p criteria.
This does not eliminate the fundamental problem of setting a threshold: a very small baseline change is indistinguishable from random variation for a very long time. But a cumulative approach should eventually detect even a tiny baseline change.
