How do I know if the difference in performance between two machine learning algorithms is significant? I built two machine learning models, A and B, and performed 10 times repeated 10-fold cross validation (CV).
The mean RMSE and $R^2$ (over all folds) for the models is:
Model A: mean RMSE=0.780, mean $R^2$=0.340
Model B: mean RMSE=0.748. mean $R^2$=0.390
So, is the difference in performance between the algorithms significant?
To check this I used the  Wilcoxon rank sum test to compare the RMSE of the 100 folds. The p-value is 0.104, so probably not significant.
However, intiuitively, a difference in $R^2$ of 0.05 over 100 folds "feels" to me noteworthy.
How can I deal with this situation? Do you have any suggestions? Instead of using 10 repeated CV, I could repeat CV 100 times, but this feels to me too much like p-hacking. What do you think?

Using cross validated estimates of out of sample performance is not advisable, since results fold to fold are correlated due to the overlapping data in each fold.  Not only that, but any tests are woefully under powered with 10 samples unless the difference in performance is extremely large, in which case there is no need for a statistical test.
To test if performance is different model to model, you may want to proceed as follows:

*

*Hold out additional data that the model has not seen.


*For each observation in this data, make a prediction with each model and compute a loss value (for RMSE, we would compute $\ell_i = \vert y_i - \hat{y}_i \vert ^2$).


*You now have samples of loss values for each model, and you can test for a difference in expected loss using a t test (note, the distribution of these loss values is irrelevant when the validation set is large enough).

