How to choose a model's hyperparameters in terms of the variance? I was solving this question about tuning hyperparameters and I don't understand how to choose the number of hyperparameters by using the training error (TE) and the validation error (VE). Define the variance as VAR=TE$-$VE.
The question is the following:
You are tuning a hyperparameter for an algorithm. The following table shows a data set with different hyperparameter, training error, and validation errors.




Hyperparameter (H)
Training error (TE)
Validation error (VE)




1
105
95


2
200
85


3
250
100


4
105
100


5
400
50





*

*Which value of H should you select based on the data?

*What H value displays the poorest training result?


The suggested answer is H=4 and H=5 respectively because H=4 minimizes the variance (variance is 5) and H=5 maximizes the variance (variance is 350).

However, I do not see how minimizing the variance is responsible for tuning the "correct" number of hyperparameters. For me it would make more sense to select the value of H which minimizes the VE for 1. (H=1) and the value of H which maximizes the VE for 2. (H=3 or H=4?).
Can someone please clarify what should be the correct choice and why?

Babado,
Your reasoning is correct.   Whenever we pick a model (to include selecting hyper-parameters), we are balancing bias and variance. The optimal hyper parameter will minimize the error created by both of these effects.
Here's an article that digs a little deeper into the subject - https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html.
Take a look at the bias-variance tradeoff (Validation Curve Schematic) graphic about halfway down.
You might want to read a little about cross-validation which is another way to identify optimal hyper-parameters.
