Machine Learning, Model Selection, Hyperparameter Tuning, Learning Curve, Cross Validation.
which we have used to select our winning model, we will get an overly optimistic estimate. Why? Let’s do a thought experiment. Assume you are given a set of black-box classifiers with the instruction to select the best-performing classifier. All classifiers are useless — they output a (fixed)