Why do we need regularization for linear least squares given that a line is the simplest model possible? In linear least squares we are trying to fit a line to data. A line is the simplest model that can be fit to the data. How is it possible for a linear model to over-fit the data? In short why do we need regularization for fitting a line to a data?
I understand that regularization improves generalization performance ie over unseen datasets and reduces over-fitting to training data. But how would regularization reduce the over-fitting issue for a least complex model (i.e linear model)?
This question was posted on stackoverflow, was redirected here. Also asked on DataScience stackexchange, but I am yet to get any answers/comments on the same.

There is an elegant theoretical reason one might want to regularize a linear model.  It is related to Dikran's answer, in that we are expressing an assumption about the weights.  In essence, L2 regularization applied to a least squares linear fit expresses a Gaussian prior assumption on weight space.  I'll show below the broad strokes of M-estimators used to derive two things:

*

*(MLE) Assume Gaussian distributed observation noise $\implies$ least squares loss gives maximum likelihood model.

*(MAP)  Assume Gaussian distribution of model weights $\implies$ L2 regularization on loss gives maximum likelihood model.

I'll leave out details for brevity, since they are available broadly already.  The point is that MSE loss and L2 regularization can be derived from first principles and simple distributional assumptions.
MLE from observation noise
In the linear regression setting, we learn model weights $\mathbf{w}$ to make scalar predictions $\hat{y}$ from samples $\mathbf{x}$ as
$$
\hat{y} = \mathbf{w}^T\mathbf{x}
$$
When one assumes the true underlying distribution is a linear combination and a Gaussian noise term,
$$
y|\mathbf{x} = \mathbf{w}^T \mathbf{x} + \mathcal{N}(0, \sigma^2)
$$
then maximum likelihood estimation (MLE) induces a mean squared error loss
$$
\mathcal{L}_{MLE}(\mathbf{w}) = \sum_{i=1}^n (\mathbf{w}^T\mathbf{x}_i - y)^2
$$
such that minimizing $\mathcal{L}_{MLE}$ produces the MLE estimate of weights.
MAP from weight distribution
Further, if one assumes a Gaussian prior distribution on the model weights $\mathbf{w}$ with each weight $w_i$ having identical variance $\nu^2$
$$
w_i \sim \mathcal{N}(0, \nu^2)
$$
then the analogous maximum a posteriori (MAP) estimation induces the L2 regularizer with regularization weight $\lambda = \frac{\sigma^2}{\nu^2}$
$$
\mathcal{L}_{MAP}(\mathbf{w}) = \sum_{i=1}^n (\mathbf{w}^T\mathbf{x}_i - y)^2 + \lambda||\mathbf{w}||^2_2
$$
such that minimizing $\mathcal{L}_{MAP}$ produces the MAP estimate of weights.
So choosing least squares loss expresses a Gaussian observation noise assumption.  And choosing L2 regularization expresses a Gaussian model weight assumption, where  $\lambda$ expresses an assumed variance ratio between observation noise and model weights.
