Which sample size equation to use for testing two onboard computers on a fleet of planes? Summary:
We have both an old tried and true computer (type #1) and brand new (type #2) on 100 airplanes processing the same raw data and want to know if they are equivalent based on the telematics output of those computers. Should we use the sample size equation for two independent samples continuous outcome
$$n_i = 2\left(\frac{Z\sigma}{ES}\right)^2$$
or sample size for matched continuous outcome
$$n = \left(\frac{Z\sigma_d}{E}\right)^2$$
Details:
We have a fleet of airplanes with onboard computers (type #1) that process raw telematics data from hardware such as the engine, brakes, GPS, altitude, etc. We are testing a new onboard computer (type #2) and hypothesize type 2’s will process that raw data identically to type #1. We run a 100 plane experiment where both type 1 and type 2 computers are present on each of the 100 planes and process the same raw data. We have flown 1 million miles so far and our CEO wants to know the sample size needed to decide if the devices are similar enough, e.g. accept or reject the null hypothesis.
What I’ve tried:
We started by using the sample size for two independent samples where the standard deviation is derived from the known standard deviation of our Type 1 computer. However, now I believe that it would be more appropriate if we used the matched sample equation for sample size because we had 100 planes with both type 1 and a type 2 computer onboard and the output are not truly independent but fit the matched testing scenario better. Since both type 1 and type 2 are present on each plane and we are comparing the outputs of type 1 vs type 2 per plane I think we need the sample size for matched samples equation where the standard deviation is based on the difference between type 1 and type 2 that we have observed so far based on the 1 million miles flown so far with both type 1 and type 2. Are both sample size equations acceptable, or do we have to pick one, and why, please?
Thank you!

Thank you for this detailed and well written explanation matmat. I can tell a lot of care and passion was involved in your response, and I was prepared to use that answer.
I ended up finding that my delta distributions were not normal and outliers were present, I should have checked that first, and wasn’t able to use the t-test. I ended up needing to use a brute force Monte Carlo bootstrap simulation with replacement because I needed to test sensitivity on number of miles per plane as well as number of planes.
Once the data was simulated with known testing outcomes I used the non-parametric Wilcoxon test to reject or accept the null. Because the data is simulated I know if the null should have been rejected or accepted and could calculate my percent of type 1 and type 2 errors. Finally I determined the acceptable sample size range by doing a sensitivity analysis based on acceptable type 1 and type 2 errors as well as varying scenarios on number of planes and miles per plane.
Thanks so much
