What exactly does PCA show that I can't figure out otherwise? First off, apologies if this is basic but I'm still learning about PCA. I am somewhat confused as to what exactly PCA can provide that I can't find out through other means. For example:

*

*Dimensionality Reduction/Redundancy: couldn't one just look at the covariance matrix? If we wanted to retain our original feature set but wanted to reduce it in some way, why not just pull out extraneous variables that are highly correlated?

*Visualization at lower dimensions/clustering: why not just run a clustering algorithm and look at in-cluster and between-cluster statistics?

The only thing I can think of that may be more useful would be if one wanted to use the latent representations or principal components for actual use somehow, like in regression. But even then I'm not sure how common this is, and I never really read about this.

"If we wanted to retain our original feature set but wanted to reduce it in some way, why not just pull out extraneous variables that are highly correlated?" PCA gives you linear combinations that represent an as high as possible amount of variation in the lower dimension. "Pulling out" some of your variables won't be as  good in terms of representing the variation, and also PCA informs you what the maximum is that can be achieved.
"Visualization at lower dimensions/clustering: why not just run a clustering algorithm and look at in-cluster and between-cluster statistics?" Looking at statistics does not give you a visualisation, and there are a lot of thing you cannot see from this, such as potential nonlinear shapes, outliers etc. PCA is a linear projection method, so the resulting image can be fairly intuitively interpreted (although of course we have to be careful because of the information loss, which is inevitable looking at higher dimensions in a low dimensional plot).
By the way, whenever I do clustering, I look at visualisations (potentially, but not only and not necessarily always PCA) to visually validate my clustering in the first place. There are many ways a clustering can go wrong, and visual validation can reveal at least some of them.
