How do I know if the difference in performance between two machine learning algorithms is significant? I built two machine learning models, A and B, and performed 10 times repeated 10-fold cross validation (CV).
The mean RMSE and $R^2$ (over all folds) for the models is:
Model A: mean RMSE=0.780, mean $R^2$=0.340
Model B: mean RMSE=0.748. mean $R^2$=0.390
So, is the difference in performance between the algorithms significant?
To check this I used the  Wilcoxon rank sum test to compare the RMSE of the 100 folds. The p-value is 0.104, so probably not significant.
However, intiuitively, a difference in $R^2$ of 0.05 over 100 folds "feels" to me noteworthy.
How can I deal with this situation? Do you have any suggestions? Instead of using 10 repeated CV, I could repeat CV 100 times, but this feels to me too much like p-hacking. What do you think?

My take on statistical testing o regression problems is different than statistical tests on classification problems. As you may know, statistical tests need sets of measures and not just single measures. So if you have a  single test set, and you calculate the RMSE or accuracy, you have a single number, and therefore no way of performing a statistical test. So you "create" many test sets by using some form of cross-validation. (k-folds, repeated k-folds and so on).
But for regression (I will discuss classification later) you do have a set of numbers - the error (or squared error) for each data in the test set. Just perform a paired t-test or paired Wilcoxon test!
You may be tense on making this decision using only one test set, which may be "unlucky" or "non-representative". But notice that the set itself is not that important, since you are using each measure of error on the data on the test set for the test. But why not use more data for the statistical test? I would suggest at most a k-fold (whatever k) - in this case, all data will contribute only one measure of error to the statistical test.
Clearly, doing a repeated k-fold (10 times a 10-fold) is p-haking since you are adding repeated data which will only decrease the p-value of the test.
TLDR: do not use RMSE on repeated k-folds - use the SE (squared error) of each data on a single k-fold.
Why not do this for classification? The problem is that the result of the classification for each data is a binary variable - the result is correct or the result is not correct. The measure for each data is a binary variable and the correct test for paired binary variables is the McNemar test which I believe (I do not have a reference for this) is a weak test. So tradition, or research end up suggesting an aggregate measure (such as accuracy, or F1, or so on) on a set of test data, and using multiple tests data to be able to have a set of numbers to perform statistical test.
Finally, I really dislike things like a 10 times repeated 10-fold against for instance F. Harrell who, for example, suggests 100-times 10-fold https://www.fharrell.com/post/split-val/. My criticism of this many repetitions of k-fold is the p-haking aspect of it - you are generating more and more data (up to $2^N$ data points where N is the size of your dataset) and that will clearly result in a low p-value for any significance test you perform (I forget the name of this theorem - infinite samples results only in p-value = 0 or p-value = 1 and this last case if the two infinite samples come from the same distribution).
