Why do we need regularization for linear least squares given that a line is the simplest model possible? In linear least squares we are trying to fit a line to data. A line is the simplest model that can be fit to the data. How is it possible for a linear model to over-fit the data? In short why do we need regularization for fitting a line to a data?
I understand that regularization improves generalization performance ie over unseen datasets and reduces over-fitting to training data. But how would regularization reduce the over-fitting issue for a least complex model (i.e linear model)?
This question was posted on stackoverflow, was redirected here. Also asked on DataScience stackexchange, but I am yet to get any answers/comments on the same.

I think part of the misunderstanding may be driven by the meaning of "model". A linear model is a set of distributions, where (for simplicity) we can consider that each distribution is represented by a line. Thus a linear model is a set (or collection) of lines - not a single line. The bigger that set is, the more complex the model is. Regularization--like removing variables from a regression--reduces the complexity of the model by making the model contain fewer lines.
This can be helpful since more complex models can be more prone to overfitting.
