Test and minimum sample size to conclude that 2 ECDFs are different Let's say I have samples $X$ from one distribution and I am now collecting new samples $Y$ from another unknown distribution. What would be the best test to check whether the samples $Y$ come from the same distribution of $X$? Should I use the Kolmogorov-Smirnov test? And how do I check that I have enough samples (as in my case the sampling step is expensive)?
As illustration: In this next scenario I can visually see after 10 samples that the distributions are very likely to be different.

However, in the following scenario, more samples are required to make the same conclusion.


As with any power and sample size determination,
the sample size to achieve given power depends on
the 'distance' between the two models being
compared.
The test statistic $D$ of the Kolmogorov-Smirnov
test is the maximum vertical distance between
the two ECDFs (plotted as step functions).
If you are using a pooled two-sample t test to
compare means of two normal distributions (having
equal variances), then there is an exact analytic
formula, using a noncentral t distribution, to
find the power for given $n$'s, significance level,
common variance $\sigma^2$ and distance $\Delta$
between means to be detected.
I know of no such formula for the K-S test, but
it is possible to use simulation to approximate the power for given sample size and significance level.
For example, suppose you want to detect the difference
between samples of size $n=10$ from distributions
$\mathsf{Gamma}(\mathrm{shape}=4, \mathrm{rate}=2)$
and $\mathsf{Gamma}(6, 3).$ Then CDFs are much
the same and ten observations from each will probably
not give good power at the 5% level.

A simulation using data from 10,000 samples
of size 10 shows the power to be uselessly small.
set.seed(1217)
pv = replicate(10^4,ks.test(
        rgamma(10,4,2), 
        rgamma(10,6,3))$p.val)
mean(pv <= .05)
[1] 0.0144  # aprx power

You would need sample sizes something like 1000 in
order to have reasonable power (say 90%) to detect the difference
between these two distributions.
set.seed(1218)
pv = replicate(10^4, ks.test(
      rgamma(1000,4,2), 
      rgamma(1000,6,3))$p.val)
mean(pv <= .05)
[1] 0.9027 # aprx power

By contrast, if you are trying to distinguish between
the distributions $\mathsf{Exp}(\mathrm{rate}=2)$ and $\mathsf{Norm}(1,1),$ then samples of size 20 might
be good enough.

A test at the 5% level with samples of size twenty has almost 80% power.
set.seed(2021)
pv = replicate(10^4, ks.test(rexp(20,2), 
                 rnorm(20,0, 1))$p.val)
mean(pv <= .05)
[1] 0.7882

Unfortunately, the K-S test is not known for good
power. If the differences between your distributions
of interest are due only to different parameter values (e.g, distinguishing between two normal distributions or two exponential distributions),
then you might find a parametric test using key parameters that
gives better results.
