What is the relationship between noise reduction and dimension reduction? My understanding is that unsupervised methods like PCA, autoencoders and K-means shape a data space such that the modified representation of the data either nicely separates different families of data points, or allows us to represent the data with using fewer dimensions.
My understanding is also closely tied to the visual image of PCA (or a non-linear manifold learning method) applied to noisy 2D data,  the data being projected onto a learnt line where after a reconstruction is possible with reduced noise. (Example from answer by amoeba).

I want to understand if there is a relationship between a noise reduction and dimension reduction. It seems to me that noise reduction will only be effective if the appropriate (In a bias-variance sense) model is used, although (potentially irresponsible) dimension reduction would be possible regardless of the model used.

*

*What is the distinction between noise reduction and dimension reduction? Do these words belong in the same sentence?

*Which fundamental mechanism in dimension reduction allows it to be capable of reducing the noise in data? What are the assumptions and trade-offs?

*Apart from an independent test set, is there a way to gauge whether dimension reduction is discarding noise, or discarding information about the true distribution of the data?

Any guidance or suggested improvements to the question will be appreciated.

This is an awesome question. So the example of PCA that you have shown projects a set of points to a vector which is along the direction of maximum covariance of the data. What is does is that,

*

*It reduces one dimension from the data, but wait what just happened due to that ??...You lost an essential component of your representation and that may just look like you got rid of noise !! Is it ?


*The principal component direction along which you are projecting only reduces the error, which is the distance of the old point and the new projected point. (Check the derivation of PCA if you did not get this)
What it does not do is that,

*

*Hey that was not only noise that you got rid from, in the real world dataset if you want want to reduce dimensions and hope that you are getting rid of noise, you are, only if you consider the definition of noise in a mathematical setting. If you think in a realistic sense, the loss of representation (as I explained in the previous point) also takes place and that combined with loss of noise together cancels each other out in terms of actual benefit. The only time you benefit from that is when you do not need so many dimensions to represent the data.

