How is adding noise to training data equivalent to regularization? I've noticed that some people argue that adding noise to training data equivalent to regularizing our predictor parameters. How is this the case?

*

*Some of the examples listed on SE discussing this topic focus more on e.g. LSTMs and SVMs, but can we do this for simpler models like a multiple linear regression?


*How will it affect our parameters' confidence intervals?


*Will there be any differences in effects choosing between the various types of white noise, e.g. Gaussian vs uniform white noise?

Overview: For linear regression, I'll show that $\ell_2$ regularization (a.k.a. ridge regression) arises from minimizing the expected squared error over random perturbations of the regressors. The distributional form of the perturbations doesn't matter beyond some minimal requirements (i.i.d., zero mean). The variance of the perturbations controls the regularization strength.
Let  $\big\{(x_i, y_i)\big\}_{i=1}^n$ be the data, with regressors $x_i \in \mathbb{R}^d$ and responses $y_i \in \mathbb{R}$. Suppose we first add random noise to the regressors, then compute predictions as a linear function of the perturbed regressors:
$$\hat{y}_i = (x_i + \delta_i)^T w$$
$w \in \mathbb{R}^d$ are the regression coefficients and the perburbations $\{\delta_i\}$ are i.i.d. random vectors with mean $\vec{0}$ and covariance matrix $\lambda I$. It's not necessary to assume that perturbations are generated from any particular parametric family.
We seek coefficients that minimize the expected squared error $L(w)$, where the expectation is taken over the random perturbations:
$$L(w) = E \left[ \frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2 \right]$$
Plug in the above expression for $\hat{y}_i$ and expand:
$$L(w) = E \left[ \frac{1}{n} \sum_{i=1}^n
  (y_i - x_i^T w)^2
  - 2 \delta_i^T w (y_i - x_i^T w)
  + w^T \delta_i \delta_i^T w
\right]$$
By linearity of expectation:
$$L(w) = \frac{1}{n} \sum_{i=1}^n \left(
  (y_i - x_i^T w)^2
  - 2 E \Big[ \delta_i \Big]^T w (y_i - x_i^T w)
  + w^T E \Big[ \delta_i \delta_i^T \Big] w
\right)$$
Note that $E[\delta_i] = \vec{0}$ and $E[\delta_i \delta_i^T] = \lambda I$ are the mean and covariance matrix of the random perturbations:
$$L(w) = \frac{1}{n} \sum_{i=1}^n \left(
  (y_i - x_i^T w)^2
  + \lambda w^T w
\right)$$
Simplify, noting that $w^T w = \|w\|_2^2$:
$$L(w) = \frac{1}{n} \sum_{i=1}^n (y_i - x_i^T w)^2 + \lambda \|w\|_2^2$$
This is the mean squared prediction error for the original (non-perturbed) data, plus a penalty on the squared $\ell_2$ norm of the coefficients. Notice that it corresponds exactly to the cost function for ridge regression, with penalty strength $\lambda$.
