How to be sure that out-of-sample results of one model are really better than those of another model? I need to solve a standard, let say "vanilla", regression problem: I have a 2D array of real-valued features $X$ and a 1D array of real-valued targets $Y$.
I choose a simple model to fit (let say a linear regressor or a decision tree) and run a k-fold cross validation of a chosen model on my data set of $X$ and $Y$. As a result I get an out-of-sample squared error that is smaller (better) than the squared error of a "constant model" (by "constant model" I mean a "model" in which always "predict" the same number independently on values of the features, which is just a mean of targets). How can I be sure that a better out-of-sample performance of my features dependent model is "real", and not just by chance? Maybe my model is statistically not better or even worse than the "constant model" and it gives better results just by chance.

Mostly people rely on Out of sample validation to be sure that their model performs better than a benchmark model. But to answer your questions you can maybe use the following hack:

*

*Step 1 : Run the model; let's say decision trees with 100 different initialisation keeping all other parameters as same. Once ran record the mean error on the out of sample data.


*Step 2 : Now you have results of 100 different models which should not vary that much as for most ML algorithms seeds just leads to slight variation.


*Step 3 : Perform a hypothesis testing on the model results as compared to benchmark model as mean (One Tailed Test)
Null Hypothesis : Model results are similar to benchmark model


*Step 4 : If you get p-value less than 0.05 or 0.01 based on how strict you want to be.
You can statistically reject the Null Hypothesis.
