Regularization and Shrinkage : Theoretical Advantages vs. Empirical Advantages I have the following question about the theoretical advantages vs. the empirical advantages of regularization (i.e. shrinkage).
As far as I understand, this is the general idea behind regularization: The "Bias-Variance Tradeoff" states that "simpler models" tend to be unable to capture complex patterns within the data and produce estimates with large biases - whereas more "complex models" tend to be able to better capture complex patterns in the data, but tend to generalize poorly to new data as they produce estimates that can greatly vary on unseen data. (note: "complexity" can be interpreted as a function of the number of parameters within the model)
As a result, regularization attempts to rectify this "Bias-Variance Tradeoff" by making complex models simpler, such that the complex models can still capture complex patterns but generalize better to unseen data. Regularization attempts to do this by "shrinking" model parameters towards 0. Doing so greatly reduces the effect these model parameters would have on the predictions, and thereby create "simplified complex model".
My Question: What proof is there that a regularized model has the (inherent) ability to overfit the data less than a non-regularized model?
I have seen some of the early proofs of the "Tikohonov Regularization Kernel" which outlines the "mathematical validity" of the Least Squares Regression Estimator when a regularization term is added to the optimization equation - but I have not seen any arguments that suggest a regularized model has the (inherent) ability to overfit the data less than a non-regularized model. Perhaps someone could argue that by virtue of the "No Free Lunch Theorem", there might in fact exist certain problems where a non-regularized model might perform better compared to a similar regularized model? I have seen countless empirical reports that display the benefits and advantages of using regularization - but I have not come across any "proofs" that suggest a "sparser model" (i.e. regularized) has the inherent ability to better perform than a similar non-regularized model.
Can someone please comment on this?
Thanks!

In the original paper by Hoerl and Kennard, in Technometrics in 1970, there is, if memory serves me well, a proof that, for $\lambda$ in some range, the ridge estimator outperforms the ordinary least squares estimator in terms of mean square error.
The problem, of course, is to ascertain what is the "right" $\lambda$ to use, but different heuristic methods have been proposed for that.
