How to test for inequality in the presence of non-independent noise? I have multiple samples which include a response time of a system. I want to test if no sample is significantly different (primarily the expected value). For two sample testing I'm using the sign test and for testing multiple ones at a time I'm using the Friedman test.
Unfortunately, the samples have non-independent noise (verified with Hoeffding's test, p-value < 1e-8). In practice that means that for samples with over 10000 observations, the sign test and Friedman test show statistically significant differences (p-value < 1e-6) for samples that were measurements of exactly the same input.
What is the recommended practice for dealing with with non-independent, non-uniform, multimodal, heteroskedastic noise for repeated measurements data?
Data acquisition
The measurements are performed as follows:

*

*First create a list of randomly ordered tuples of tests to perform (e.g., given three inputs, A, B, and C, it could be something like ABC, CBA, CAB, BCA, etc.)

*Run the tests in that random order (e.g. send input A, wait for reply, send B, wait for reply, send C, wait for reply, send C, wait for reply, send B, wait for reply, etc.)

*

*The inputs are sent by a Python application over a regular TCP connection, the "protocol" is just connect, send query, wait for response, close connection



*Have a system running in the background that monitors the communication, noting the time between query and response, saves that as a list (continuing the example: 68028 ns, 69667 ns, 67971 ns, 68535 ns, 69458 ns, 67767 ns, 68335 ns, ...)

*

*This is done by tcpdump



*Combine the knowledge of the ordering of the tests to the noted times to get measurements for specific tests (continuing the example, for A I then get 68028 ns, 67767 ns, 67822 ns, ..., for B I get 69667 ns, 69458 ns, 68314 ns, ..., and for C I get 67971 ns, 68535 ns, 68335 ns, ...)

Data example
Example scatter plot of a pair of samples (axes are in
seconds):


More data
I did an additional run with N=12800, where all the probes were sending the exact same values, but were generated by 2 different objects. (Same classes, initialised with the same values, but separate instances).
The first 4 were generated by one object, while the last 3 were generated by a second object.
I've then bootstrapped confidence intervals for the calculated medians of differences:

Reinterpreting the data from question
Neither looking at the PACF of the differences between B and A:

Nor running tests like the Wald-Wolfowitz runs test strongly point to lack of independence of the differences between samples (p-value=0.017).
But if we look at the windowed median (window=1000, step=1):

It's fairly clear that the false positive in the sign test isn't caused by one large single excess or periodic excess in one sample over the other, but rather in systematic departure from the zero median.
Hypothesis
The false positives are caused by small, but constant differences in how the different instances of objects are handled in the python interpreter.
This in turn has effect on the delay between when the connection is opened and when the data over it is sent. And that affects how quickly the server responds.
In other words, it's the effect of PYTHONHASHSEED and/or ASLR.
Solution
The solution is to send probes from new processes, so that any effect from PYTHONHASHSEED and/or ASLR is the same for different instances.
Verification
I've thus modified the execution of the tests so that any single process isn't used to send more than 100 probes.
I've then repeated the experiment few dozen times with large sample sizes (>100k) and not only are the false positives gone, the distribution of the p-values of those tests is perfectly uniform.
