CV for Model selection - Why is significance testing not needed? Say we have a model and some hyper-parameter with L values, and our goal is model selection. A k-fold CV outputs L accuracies (each accuracy is an average over K values). The best model corresponds to the highest accuracy.
When comparing the L accuracies in order to select the best model, why don't we do any significance testing?
My guess is that there is no cost in wrongly rejecting the null hypothesis (== saying one model is better than the other while they are equivalent), therefore there is no harm in selecting the model with the highest accuracy. The worse that can happen is that all L models are equivalent, but even if that's the case we still want to select one of them, so there is no harm in selecting the highest accuracy (it's as arbitrary as anything else)

Summary: IMHO, it would be good to do significance testing for model selection. (But as @Christian Hennig already pointed out, it's actually not so easy to do)
Or rather: we'd need to get a far more realistic idea of the random uncertainty our hyperparameter tuning is subject to.
Whether that takes the shape of significance testing, Bayesian model averaging, or more sophisticated cross validation, any other for or a combination thereof I don't care.
But I do think we have overall a serious problem with overfitting. (Which I think also has strong links to the reproducibility crisis in the life sciences)

@Dikran Marsupial points out that hyperparameters are not inherently different from other parameters of a model. The distinction is mainly a question of implementation convenience, leaving parameters that are difficult to fit correctly in the general case to the care of the user.
This is also basically the approach that has been used for many decades in classical statistics: e.g. for a linear model, we know how to calculate a least squares fit on given data. It is the genuine task of the one who models some data to specify the model terms, including model complexity, feature selection (which variates to include/drop) and feature construction (e.g. polynomial terms).
to be continued
