What is the relationship between noise reduction and dimension reduction? My understanding is that unsupervised methods like PCA, autoencoders and K-means shape a data space such that the modified representation of the data either nicely separates different families of data points, or allows us to represent the data with using fewer dimensions.
My understanding is also closely tied to the visual image of PCA (or a non-linear manifold learning method) applied to noisy 2D data,  the data being projected onto a learnt line where after a reconstruction is possible with reduced noise. (Example from answer by amoeba).

I want to understand if there is a relationship between a noise reduction and dimension reduction. It seems to me that noise reduction will only be effective if the appropriate (In a bias-variance sense) model is used, although (potentially irresponsible) dimension reduction would be possible regardless of the model used.

*

*What is the distinction between noise reduction and dimension reduction? Do these words belong in the same sentence?

*Which fundamental mechanism in dimension reduction allows it to be capable of reducing the noise in data? What are the assumptions and trade-offs?

*Apart from an independent test set, is there a way to gauge whether dimension reduction is discarding noise, or discarding information about the true distribution of the data?

Any guidance or suggested improvements to the question will be appreciated.

What you see on the plot is not exactly a noise reduction. What is shown is the predictions made using only the first component of PCA. Using the first component of PCA leads to reducing multi-dimensional data to a single dimension. The new representation is less rich in information because to create it we disregarded the information that was represented in the other components. This can lead to noise reduction, but this also removes the non-noise, relevant information.
Moreover, it doesn't necessarily need to remove the noise. As an example, let's use a thought experiment: if you used PCA on completely "random" data (noise-only), the first component (or components) would contain something, as the algorithm forces them to be something. The algorithm in such a case wouldn't do anything to remove the noise, it would rather "overfit" to some bogus patterns.
Also keep in mind that what you see for PCA, or other dimensionality reduction algorithms, would also hold for any other machine learning algorithm that doesn't completely overfit the data. If the predictions by the machine learning model won't perfectly fit the data (they usually won't), it would produce predictions that are "simpler" and "smoother" than the observed data, because the algorithm needed internally to create a simplified representation of the data (it has fewer parameters than data points). Again, this doesn't necessarily remove the actual noise and whatever algorithm you would use for noise reduction, this is something to be verified when validating the model.
