Why do we need regularization for linear least squares given that a line is the simplest model possible? In linear least squares we are trying to fit a line to data. A line is the simplest model that can be fit to the data. How is it possible for a linear model to over-fit the data? In short why do we need regularization for fitting a line to a data?
I understand that regularization improves generalization performance ie over unseen datasets and reduces over-fitting to training data. But how would regularization reduce the over-fitting issue for a least complex model (i.e linear model)?
This question was posted on stackoverflow, was redirected here. Also asked on DataScience stackexchange, but I am yet to get any answers/comments on the same.

Great question! The need for regularization always depends on your sample size.
Imagine you do not have a lot of data, just three samples. I plotted three possible linear regression lines. The red one does not use regularization; for the green one, regularization on the slope parameter is used, and the blue regression has very strong regularization on the slope parameter.

Which one is the best model? We don't know. That will only become clear as more data is collected. But regularized models produce more conservative estimates, which often work better in practice.
That said, if you really only have the case of a simple linear regression (one input variable, like in my example), you will often have enough data to use an unregularized model.
But keep in mind that linear regression also works for multiple input variables. In that case, the model will have $p+1$ parameters if you have $p$ features. You need regularization if $p$ is large compared to your sample size.
In summary, a model is never a priori complex or simple, it always depends on the data you want to use the model for.
