PCA as a Cure for the Curse of Dimensionality I would like some clarification as to how principal component analysis mitigates the Curse of Dimensionality problem. My particular interest is in curbing overfitting in my modelling, or more specifically parameter count. If I use all 30 of my features I will have a model with 30 parameters: this is too large a number for my sample size, overfit is almost guaranteed. I am told that I should rather build my model with the first 3 principal components of my feature set and thus have only a 3 parameter model, and apparently mitigate my overfitting problem. But then I have computed 30x30 elements for my eigenvector matrix and 3 parameters for my model, I have fitted 900+3 parameters to the data. Now I have gone from a model with a maximum parameter count of 30 to a model with 903 parameters. How have I evaded the Curse of Dimensionality? It is really not obvious to me. An additional issue is the high variance of the elements of the eigenvector matrix, I have noted that relatively small changes to the feature data cause considerable variation in these elements, sometimes even changing signs. They are more unstable than the parameters of the model that I am trying to fit.

In a way, PCA does not use the outcome you are trying to model/predict, i.e. it is an unsupervised technique. From that perspective, its parameters are not parameters that get trained in your supervised model. Of course, using PCA for dimensionality reduction is not in any way guaranteed to preserve "the signal" for the outcome of interest that may be in the data (see e.g. this previous question for a discussion). I.e. it may well be preferable to select the most important variables based on subject-matter expertise, if there is a decent amount of prior knowledge. There's of course also other techniques/alternatives to PCA (e.g. various variants of PCA, UMAP, t-SNE, training a denoising autoencoder on the features and so on).
However, a lot may also depend on your goals. Are you trying to interpret the model coefficients (if so, PCA does make that harder), are you trying to create a prediction model that is meant to achieve a certain level of performance (if so, interpretability of PCA may be less of a concern, but it may also be even more of a problem to be working with too little data), or are you trying to do something else?
