Curse of dimensionality: How PCA improves my model? After having read about the curse of dimensionality, I have been looking into a filtered version of the Superconductivity dataset.
The number of dimensions is 81, so initially I thought that reducing the dimensions using PCA could help to improve the prediction power. (The majority of the articles I have read about the curse of dimensionality speak about dimensions greater than 10)
Surprisingly, I found the opposite:
# Using caret library and a loop to iterate 
# over the number principal components to use:
for (pca_component_i in seq(5, 80, by = 5))
    {
        fitControl <- trainControl(
            preProcOptions=list(pcaComp = 
              pca_component_i),
            method = "repeatedcv",
            number = 4,
            repeats = 2,
            summaryFunction=defaultSummary,
            verboseIter = FALSE
        )
    
model1 <- train(train_X, train_y, 
            method = "glm", 
            trControl = fitControl,  
            preProc = c("pca"),
            metric =  "RMSE"
        )
print(RMSE(predict(model1, newdata=test_X), 
      test_y))
    }
        
    [1] 20.42709
    [1] 18.36957
    [1] 15.0958
    [1] 11.05231
    [1] 2.58729
    [1] 1.250649
    [1] 0.3855323
    [1] 0.2942177
    [1] 0.1621667
    [1] 0.1147847
    [1] 0.03312724
    [1] 0.02567603
    [1] 0.01053921
    [1] 0.007634504
    [1] 0.001618915
    [1] 7.672395e-06

On the other hand, it also looks pretty obvious behavior cause more principal components explain more variance. So,

*

*What am I misunderstanding?

*Is there any possible situation where reducing the dimensionality of the data lead to a increase of the prediction power?


With over 20,000 cases in the linked data set, your 81 predictors probably aren't getting you into trouble with "the curse of dimensionality" for this application. From pages 265 - 266 of "An Introduction to Statistical Learning," second edition (ISLR) on this topic:

In general, adding additional signal features that are truly associated with the response will improve the fitted model, in the sense of leading to a reduction in test set error. However, adding noise features that are not truly associated with the response will lead to a deterioration in the fitted model, and consequently an increased test set error. This is because noise features increase the dimensionality of the problem, exacerbating the risk of overfitting (since noise features may be assigned nonzero coefficients due to chance associations with the response on the training set) without any potential upside in terms of improved test set error.

The associated example (Figure 6.24) had 100 observations, with 20 features truly associated with outcome and additional random features in LASSO modeling. In this situation they started to get into trouble when the number of candidate predictors got above 20 or so, or a case/predictor ratio of about 5.
In contrast, you have about 250 cases per predictor. One typically gets into problems when this ratio is down close to the single digits or, as can happen for example in bioinformatic analyses, you have more potential predictors than cases so that you must reduce the effective dimensionality of the predictor set for regression. As the example from ISLR demonstrates, even LASSO for penalization might not help with a very large predictor set if most predictors are unassociated with outcome.
