CV for Model selection - Why is significance testing not needed? Say we have a model and some hyper-parameter with L values, and our goal is model selection. A k-fold CV outputs L accuracies (each accuracy is an average over K values). The best model corresponds to the highest accuracy.
When comparing the L accuracies in order to select the best model, why don't we do any significance testing?
My guess is that there is no cost in wrongly rejecting the null hypothesis (== saying one model is better than the other while they are equivalent), therefore there is no harm in selecting the model with the highest accuracy. The worse that can happen is that all L models are equivalent, but even if that's the case we still want to select one of them, so there is no harm in selecting the highest accuracy (it's as arbitrary as anything else)

There isn't usually a genuine statistical distinction between hyper-parameters and parameters, it is normally just a matter of computational convenience.  There is a computationally efficient method of optimising the parameters for fixed values of the hyper-parameters, so it is sensible to use it.  We call the parameters for which there is no computationally efficient algorithm "hyper-parameters" and use cross-validation to tune them instead, which is often computationally expensive.  So the main reason we don't need statistical hypothesis tests for hyper-parameters is that we don't usually need them for parameters either.
However, it is possible to over-fit the cross-validation error when tuning the hyper-parameters, just as you can over-fit the training data when optimising the parameters.  This can result in a model that under-fits the training data as well as over-fitting it.  There are a number of practical solutions, such as to regularise the hyper-parameters, another is to tune the hyper-parameters and then backtrack to the point where the cross-validation performance is not statistically distinguishable from that of the best set of hyper-parameters, so you can use NHSTs in model selection if you want to (sort of "early stopping").
