Curse of dimensionality: How PCA improves my model? After having read about the curse of dimensionality, I have been looking into a filtered version of the Superconductivity dataset.
The number of dimensions is 81, so initially I thought that reducing the dimensions using PCA could help to improve the prediction power. (The majority of the articles I have read about the curse of dimensionality speak about dimensions greater than 10)
Surprisingly, I found the opposite:
# Using caret library and a loop to iterate 
# over the number principal components to use:
for (pca_component_i in seq(5, 80, by = 5))
    {
        fitControl <- trainControl(
            preProcOptions=list(pcaComp = 
              pca_component_i),
            method = "repeatedcv",
            number = 4,
            repeats = 2,
            summaryFunction=defaultSummary,
            verboseIter = FALSE
        )
    
model1 <- train(train_X, train_y, 
            method = "glm", 
            trControl = fitControl,  
            preProc = c("pca"),
            metric =  "RMSE"
        )
print(RMSE(predict(model1, newdata=test_X), 
      test_y))
    }
        
    [1] 20.42709
    [1] 18.36957
    [1] 15.0958
    [1] 11.05231
    [1] 2.58729
    [1] 1.250649
    [1] 0.3855323
    [1] 0.2942177
    [1] 0.1621667
    [1] 0.1147847
    [1] 0.03312724
    [1] 0.02567603
    [1] 0.01053921
    [1] 0.007634504
    [1] 0.001618915
    [1] 7.672395e-06

On the other hand, it also looks pretty obvious behavior cause more principal components explain more variance. So,

*

*What am I misunderstanding?

*Is there any possible situation where reducing the dimensionality of the data lead to a increase of the prediction power?


Higher order PC components need not be the most important features.
PCA only helps you when the higher variation components are meaningful in explaining the variation in the $Y$ variable.

*

*Indeed, Sometimes PCA helps you in finding meaningful correlation patterns in the features and this could indicate some relevant principle that might align with the $Y$ variable.

*But, the pattern can be something else and may not need to explain $Y$.

see for instance the image from this question correlation of features and target in predicting red wine quality in machine learning

In the figure above, the difference between the groups is very subtle and does not correspond with the large variability in the parameters.
Within group variability
The situation of a low explanation power by higher order principal components occurs when some parameters are correlated and have a large variance, which creates the PCA components, but the differences in the variable $Y$ of interest is perpendicular to this.
This may happen if the features can have a large variability given the same value for the parameter of interest (e.g. given the same group or given the same critical temperature). Especially when the variation within the group follows some relationships (and is correlated), then the PC1 component will be more relating to within-group variation than between-group variation. In the example above this relationship is between sugar and alcohol which are negatively correlated (because sugar is turned into alcohol)
Example plot with your data
In a lot of physical/chemical experiments parameters are varied more or less along curves (defined by some physical law with a bit of variability). An example is below using your data set where we only focus on two features (for simplicity of plotting). The experimenter has the ability to change atomic mass and density by changing the composition, but they are very much correlated, and not every combination is possible. The variation is mostly along a curve.
The PC1 or other higher-order PC components are gonna be aligned tangent to those curves, but the relevant parameter of interest might be varying perpendicular to this.

The dataset is especially having this issue because it has the same parameter repeated several times in a slightly different way, like geometric mean and arithmetic mean. Below you see this for a plot of the atomic mass. You see this green region in the middle (the high temperature conductors), which is stretched out along a line. This probably relates to components that are a particular ideal mixture of small and large atomic mass atoms such that the geometric mean is along a line about 0.75 times the arithmetic mean (the pure compositions are along the line geometric mean = arithmetic mean).

