Feature selection- feature that most of its values are equal. How can I know if to drop it? If I have a feature that most of its values are the same, how can I know if to drop it or to keep it?
First of all, most of the model's features have low variance. Secondly, maybe the observations that have different values in this feature get different values in the dependent variable.
Is there any method or heuristic that can help me decide if should I keep this feature?
For example the next feature:


The case when this feature is most useful is when you have a classification task and the values of this feature perfectly match the outcome you're trying to predict. The case when this feature is least useful is when it has no relationship to the outcome.
Without knowledge of the outcome, we can't know whether or not this feature helps predict the outcome. But there are two tests that can help you eliminate unhelpful features and redundant features a priori. Because these tests are not aware of the outcome, they are not sufficient to establish that a feature is useful; they can only be used to discard features that do not add any information.

*

*If the feature is a constant (equiv., the variance of the feature is exactly 0). In a regression setting, this coincides with (2) when there are complete dummy bases and an intercept. In other settings, like kernel regressions or tree-base models, the constant feature will simply not add any information to the model.

*There is a set of features that is a linear combination of another set of features. The classic example is the so-called "dummy variable trap," where representing two or more categorical features as dummy variables create a design matrix with linearly dependent columns. But linearly dependent columns can arise in other forms, such as storing the same measurements in two different scales that are linear functions of each other (e.g. meters and kilometers, or Fahrenheit and Celsius).

Detecting (1) is simple: you just need to count the number of distinct values in a column. (The only sharp corner here is making allowances for roundoff error in floating point arithmetic.)

*

*The plot in OP's question shows that this feature is not constant (though it is predominantly a single value), so we know that we cannot eliminate the feature due to (1).

Detecting (2) is more involved, but the gold-standard way to determine the rank of a matrix $M$ is to compute its SVD. The number of positive singular values is the rank of $M$; truncating to the $k$ left singular vectors corresponding to the $k$ positive singular values gives a new matrix which has linearly independent columns.

*

*We don't know anything about the rest of the features, so the feature in OP's histogram might or might not be eliminated by (2).

If we broaden the scope of tools to include feature-selection methods that are aware of the outcome, we have many more tools in the toolbox. Some examples are:

*

*chi-square tests of independence

*boruta

*lasso

*information gain and mutual information methods.

