When we do cross validation what are we cross validating? First the definition. Cross validation: With only the training dataset, we estimate what the test error would be.
Obviously we want to do whatever we can to get the lowest test error.
However, which of the following are we actually cross validating in this process?

*

*Feature selection within a model (eg. which features are important/have high variance/contributes to the model vs. which features are irrelevant and should not be included in the model)


*Model selection (pick between multiple models) (eg. logistic reg vs. random forest vs. XGBoost)


*Hyperparameter selection (within a model)


*All of the above

There is no restriction, so it can be all of the above. However, it is usually expensive to do all and we choose the most important things to tune.
