How do I know if the difference in performance between two machine learning algorithms is significant? I built two machine learning models, A and B, and performed 10 times repeated 10-fold cross validation (CV).
The mean RMSE and $R^2$ (over all folds) for the models is:
Model A: mean RMSE=0.780, mean $R^2$=0.340
Model B: mean RMSE=0.748. mean $R^2$=0.390
So, is the difference in performance between the algorithms significant?
To check this I used the  Wilcoxon rank sum test to compare the RMSE of the 100 folds. The p-value is 0.104, so probably not significant.
However, intiuitively, a difference in $R^2$ of 0.05 over 100 folds "feels" to me noteworthy.
How can I deal with this situation? Do you have any suggestions? Instead of using 10 repeated CV, I could repeat CV 100 times, but this feels to me too much like p-hacking. What do you think?


is the difference in performance between the algorithms significant?

$H_0$: Model A and B have the same generalization error
$H_1$: Model B has a lower generalization error than model A
In cross-validation, the out of sample error approximates the generalization error. When you're doing $n$ repeated $k$-fold cross validation, you're producing $n$ different models (assuming different random seed and training sets due to different $k$-fold split). Therefore you don't compare the models, you're comparing the algorithms and hyperparameters.
Rather, let models A and B be produced once by a $k'$-fold cross validation with $k'=nk$. You obtain $k'$ out of sample errors for model A and B. You can then apply Mann-Whitney U test to know how significant the difference in generalization error is.
Assuming you don't set an alpha level to reject the null hypothesis $H_0$ and you get $p=0.104$, the roughs guidelines are:

*

*If $p > .10$ → “not significant”

*If $p ≤ .10$ → “marginally significant”

*If $p ≤ .05$ → “significant”

*If $p ≤ .01$ → “highly significant.”

Now what does it means in practice? If you were to choose between the two models, rather have the one with the lowest estimated generalization error. But should you spend time changing an algorithm in production for such difference? Not if you don't expect a lot of business revenue generated by this difference.
