How is adding noise to training data equivalent to regularization? I've noticed that some people argue that adding noise to training data equivalent to regularizing our predictor parameters. How is this the case?

*

*Some of the examples listed on SE discussing this topic focus more on e.g. LSTMs and SVMs, but can we do this for simpler models like a multiple linear regression?


*How will it affect our parameters' confidence intervals?


*Will there be any differences in effects choosing between the various types of white noise, e.g. Gaussian vs uniform white noise?

The basic concept behind regularization is that we start with our Bayesian prior for the coefficients being a decreasing function of the magnitude of the coefficient. That is, the Bayesian prior for the coefficient being large is smaller than the prior for the coefficient being small. If the basic loss function gives a large estimate for the coefficients, our final estimate is a balance between the Bayesian prior giving more weight to smaller estimates, and the data providing evidence for larger ones.
If the random noise corresponds to coefficients being $0$, then it will also pull our final estimates towards being smaller; our actual data saying the coefficients are large will have to compete with the random noise saying the coefficients are small. Another way of looking it is that if we add noise that is generated according our priors, then that will decrease the degree to which our data causes our final estimates of the coefficients to deviate from our priors. Both regularization and random noise are ways of increasing the effects of our priors on our final estimates.
