Why do we choose the hyperparameters that gives the lowest validation error? Do we assume that it also gives the lowest generalization error? The usual way of selecting hyperparameters is to tune it on the validation set and select the hyperparameters that gives the lowest validation error (Lets assume the validation sample is large so we dont get a "lucky sample", or we can do cross validation). We then use the test set to get an estimate of the generalization error.
In doing so, are we assuming that the hyperparameters that gives the lowest validation error also gives the lowest generalization error? Has this been proven or do we just base this off of intuition?


In doing so, are we assuming that the hyperparameters that gives the
lowest validation error also gives the lowest generalization error?

Yes. Validation represents the test set in HP tuning process. And similarly even the test set represents all the unseen data outside your dataset. The success in the test set is not the true generalization error but an estimate, because that would be measured with the data distribution.

Has this been proven or do we just base this off of intuition?

This is definitely intuitive, but at the same time based on theory. Assuming iid samples in the validation and test sets, the expected generalization error can be estimate simply via monte carlo:
$$\mathbb E[ \mathcal L(Y, \hat Y=\hat f(X))]\approx \frac{1}{n_v}\sum_{i=1}^{n_v} L(y_{v,i}, \hat f(x_{v,i}))$$
where $\mathcal L$ is the loss function, $n_v$ is the size of the validation set, and $(x_{v,i}, y_{v,i})$ are samples in the validation set. Note that the true generalization error is also estimated by the test set. So, a downward trend in the generalization error estimated by the validation set is a good sign for the true generalization error to decrease.
