Why we do train/val/test split in ML models, but not in regularized linear regression? There is one thing in data science that I cannot understand. When we have algorithms like Random Forest, Gradient Boosting, Neural Networks, we split our data into three parts - train (to train our model), validation (to choose best hyperparameters), and test (to get most accurate out of sample error).
In standard OLS linear regression it's obvious that we won't split our data to training, validation and testing, but only training and testing, since there is no hyperparameters to optimize. However, the game changes when we consider regularization, which implies need of estimation of hyperparameters associated with certain regularization (L1, L2 or Elastic Net). I've never seen in my life, that someone would split data into three parts, to search on the validation set for the best hyperparameters for regularization. In contrast everyone does it on test set.
But this is a little problematic isn't it? Observe, that if you choose regularization hyperparameters on test set, then you don't posses any accurate out of sample result (and this is argument, why you should provide a validation set).
Could you please explain to me this phenomenon?

It’s very common to use a validation set or a cross-validation scheme (k-fold CV, nested CV) to select the regularization hyper parameter. The software for regularization sometimes abstracts this step away. Glmnet is the gold standard for regularization in R. You can choose the penalty parameter(s) with cv.glmnet(). The cv in cv.glmnet() refers to cross validation. In Python, GridSearchCV,RandomSearchCV, or EstimatorCV (as suggested by @BenReiniger), can be used to find the best penalty hyperparameter of a lasso/ridge/elastic-net estimator.
However, for linear ridge regression models you can calculate leave-one-out cross validation with a closed form expression. You don’t need to go through the process of actually doing leave-one-out cross validation. This is called generalized cross validation.
