How can a feature that when removed, does not affect the model's performance not be declared unimportant? In the paper on the Boruta algorithm, there is a statement that is unclear to me (highlighted in black).

The all-relevant problem of feature selection is more difficult than usual minimal-optimal one.
One reason is that we cannot rely on the classification accuracy as the criterion for selecting
the feature as important (or rejecting it as unimportant). The degradation of the classification
accuracy, upon removal of the feature from the feature set, is sufficient to declare the feature
important, but lack of this effect is not sufficient to declare it unimportant.

My questions are:

*

*How can a feature that when removed, does not affect the model's performance not be declared unimportant?


*Is there a scenario that can exemplify this case?

One immediate example which comes to my mind would be a case where predictors are highly co-linear. If you have two covariates which are highly correlated, removing one of them will improve the performance of your model. Yet, it does not mean that the removed covariate was not relevant to predict your outcome variable. Here is an example with R from there :
library(faraway)
hip_model = lm(hipcenter ~ Age + Arm + Ht + HtShoes, data = seatpos)
# Adjusted R2 = 0.62
hip_model_small = lm(hipcenter ~ Age + Arm + Ht, data = seatpos)
# Adjusted R2 = 0.63
hip_model_deletedfeat = lm(hipcenter ~ HtShoes,data=seatpos)
summary(hip_model_deletedfeat)

Running this code, you will see that removing the HtShoes variable in your model will effectively increase the accuracy of your model but it does not mean that HtShoes is irrelevant since, if you regress your outcome variable on this covariate alone, you see that the relationship is statistically significant.
