Curse of dimensionality: How PCA improves my model? After having read about the curse of dimensionality, I have been looking into a filtered version of the Superconductivity dataset.
The number of dimensions is 81, so initially I thought that reducing the dimensions using PCA could help to improve the prediction power. (The majority of the articles I have read about the curse of dimensionality speak about dimensions greater than 10)
Surprisingly, I found the opposite:
# Using caret library and a loop to iterate 
# over the number principal components to use:
for (pca_component_i in seq(5, 80, by = 5))
    {
        fitControl <- trainControl(
            preProcOptions=list(pcaComp = 
              pca_component_i),
            method = "repeatedcv",
            number = 4,
            repeats = 2,
            summaryFunction=defaultSummary,
            verboseIter = FALSE
        )
    
model1 <- train(train_X, train_y, 
            method = "glm", 
            trControl = fitControl,  
            preProc = c("pca"),
            metric =  "RMSE"
        )
print(RMSE(predict(model1, newdata=test_X), 
      test_y))
    }
        
    [1] 20.42709
    [1] 18.36957
    [1] 15.0958
    [1] 11.05231
    [1] 2.58729
    [1] 1.250649
    [1] 0.3855323
    [1] 0.2942177
    [1] 0.1621667
    [1] 0.1147847
    [1] 0.03312724
    [1] 0.02567603
    [1] 0.01053921
    [1] 0.007634504
    [1] 0.001618915
    [1] 7.672395e-06

On the other hand, it also looks pretty obvious behavior cause more principal components explain more variance. So,

*

*What am I misunderstanding?

*Is there any possible situation where reducing the dimensionality of the data lead to a increase of the prediction power?


It is a mistake to go into your problem thinking that fewer parameters is better. Remember the bias-variance decomposition. You are reducing the parameter count to lower the variance, but that doesn’t necessarily come with a small increase to the bias. It is totally possible that, by omitting relevant features, you bias the model too much to result in a decrease in MSE.
This answer of mine does not address PCA, but it does address the idea of decreasing the number of parameters in an attempt to improve performance and why that need not help.
