Evaluating Supervised Model Performance Against a Baseline My question is regarding how I can interpret the performance of a supervised ML task relative to a baseline estimator.
I have run a supervised ML as a regression, and used K-fold CV to evaluate performance. This has given me an array of RMSE scores (one for each fold), which I am hoping to compare to another array of RMSE scores obtained from the baseline.
Would a simple t-test be appropriate here? Or are there better ways to approach the problem? If a t-test is appropriate, should it be considered a paired or unpaired test? The baseline, which is simply the training mean repeated for every prediction, is calculated within the same fold as the model, which makes me think it should be paired.... but I'm not sure.


Would a simple t-test be appropriate here?

Doubtful.  A t test based on the CVRMSE scores would be underpowered and not account for the precision of estimates (I'd be fairly confident in the RMSE score of it came from a holdout set of very large size).
I'm not sure how great this approach would be, but you could have a sort of "evaluation" dataset and perform a t test on the errors.
Take your two models and fit them on the training data.  Use each model to make predictions on the evaluation set.  Because this is a supervised test, you can get the errors for each model and hence the loss values for each model.  Since you are using RMSE, the loss value $\ell_i$ is just the absolute value of the error $e_i$,  $\ell_i = \sqrt{e_i^2} = \vert e_i \vert $.  It might be sensible to do a t test on the loss values, the hypothesis being that the first model and the second model have equal expected loss.  I imagine you want a model which performs better than the baseline, so you can do a one tailed test in this case.
