Effect of increaing number of features on support vector regression with a linear kernel I am trying to train and optimize a linear kernel support vector regression while analyzing the effect of increasing the number of features used to train the model on the model performance.
The number of features used to train the model is increased according to their pearson's correlation coefficient value. I am doing nested cross validation for each subset of n features.
The script is the easiest way to explain exactly what I am doing
The variable "all_features_names" has names of all variables arranged in descending order according to the value of the correlation coefficient of each feature
n_features_max = 25
for n_features in range(1,n_features_max+1):
    subset_features_names = all_features_names[0:n_features]
    if n_features == 1:
        X = features_datatable[subset_features_names].to_numpy().reshape(-1, 1)
    else:
        X = features_datatable[subset_features_names].to_numpy()
    y = target.to_numpy()

    cv_outer = KFold(n_splits=4, shuffle=True, random_state=1)
    train_error = []
    test_error = []
    for train_index, test_index in cv_outer.split(X, y=y):
        X_train = X[train_index,:]
        y_train = y[train_index]
        X_test = X[test_index,:]
        y_test = y[test_index]

        # Cross validation grid search
        cv_inner = KFold(n_splits=4, shuffle=True, random_state=1)
        if n_features == 1:
            parameters = [{'C': C_range, 'epsilon':epsilon_range}]
            model = SVR(kernel='linear')
        else:
            parameters = [{'estimator__C': C_range, 'estimator__epsilon':epsilon_range}]
            norm = StandardScaler()
            estimator = SVR(kernel='linear')
            model = Pipeline(steps=[('normalization',norm),('estimator',estimator)])
        scorer = 'neg_mean_absolute_percentage_error'
        search = GridSearchCV(model, param_grid=parameters, cv=cv_inner, scoring=scorer, n_jobs=-1, return_train_score=True)
        search.fit(X_train, y_train)
    
        # optimum model performance
        train_error.append(mean_absolute_percentage_error(y_train,search.predict(X_train))*100)
        test_error.append(mean_absolute_percentage_error(y_test,search.predict(X_test))*100)
             
    train_error_mean = np.mean(train_error)
    train_error_std = np.std(train_error)
    test_error_mean = np.mean(test_error)
    test_error_std = np.std(test_error)
    features_model_performance_dt.loc[n_features-1,'subset_features_names'] = subset_features_names.to_list()
    features_model_performance_dt.loc[n_features-1,'train_error'] = train_error_mean
    features_model_performance_dt.loc[n_features-1,'train_error_std'] = train_error_std
    features_model_performance_dt.loc[n_features-1,'test_error'] = test_error_mean
    features_model_performance_dt.loc[n_features-1,'test_error_std'] = test_error_std


In figure(1) I am plotting the train and test error vs the number of features used, while in figure(2). I am plotting the difference between them. That difference should indicate if the model is overfitting or not. What I can't understand is why is the level of overfitting increasing and then decreasing again. I was expecting that it would be either increasing as the number of features increases or it will not increase because a linear kernel support vector regression is supposed to handle a high number of features.
I know that feature selection should be embedded in the inner loop, but a requirement of my project is to determine exactly which features are most useful, that's why I am doing it prior to model optimization.
I hope I was able to explain it clearly, please let me know if anything is not clear.
Thank you!



Maybe cause the dependability lies on what these features extract, and not just the number of them. At point 13 on the second graph for instance, the features that you have included were more discriminative or perhaps when combined with the previous ones enabled more "discriminance".
