Feature subset selection decision rule I want to select a feature subset. I know that every feature in this dataset is informative to some level, due to domain knowlege. So in theorey I should use all features to maximize the negative mse between the target and a linear combination of the features.
I get the following result, when I'am doing a RFECV:
Code:
from yellowbrick.model_selection import RFECV
from sklearn.model_selection import KFold
import sklearn

seed = 42
cv = KFold(n_splits=5, shuffle=True, random_state=seed)

model = sklearn.linear_model.LinearRegression()
visualizer = RFECV(model, cv=cv, step=1, scoring='neg_mean_squared_error')
visualizer.fit(X_train, y_train)  # Fit the data to the visualizer

visualizer.show()  # Finalize and render the figure

Output:

As you can see, the negative mse does not change very much, when I keep 80 to 40 features. When I use 40 to 10 features, the mse increases slightly. If I use below 10 features, it increases sharpely. Following the RFECV keeping 72 out of 80 is optimal. Im other datasets the RFECV tells me that I should keep all features.
Because of costs that are connected with keeping features, I want to choose as few as possible. When I look at the plot, I would choose ~40 features according to my intuition.
The question is: Is there a quantitative decision rule for this that supports my intuition?

For those who are interessed: As suggested in Hastie et. al., The Elements of Statistical Learning, 2nd Edition: p. 61, 244 (Corrected 12th printing - Jan 13, 2017). one could use the "one-standard-error"-rule, where you pick the most parsimonious model whose error is no more than one standard error above the error of the best model.
This rule of a thumb is implemented in the Sequential Feature Selector from mlxtend: http://rasbt.github.io/mlxtend/api_subpackages/mlxtend.feature_selection/#sequentialfeatureselector
