What is the main purpose of Feature Selection? I have a small medical dataset (200 samples) that contains only 6 cases of the condition I am trying to predict using machine learning. So far, the dataset is not proving useful for predicting the target variable and is resulting in models with 0% recall and precision, probably due to the scarcity of the minority class.
However, in order to learn from the dataset, I applied Feature Selection techniques to deduct what features are useful in predicting the target variable and see if this supports or contradicts previous literature on the matter.
When I reran my models using the reduced dataset, this still resulted in 0% recall and precision. So the prediction performance has not improved using feature selection. But the features returned by the applying Feature Selection have given me more insight into the data.
So my question is, is the purpose of Feature Selection:

*

*to improve prediction performance

*or can the purpose be identifying relevant features in the prediction and learning more about the dataset

So in other words, is Feature Selection just a tool to achieve improved performance, or can it be an end in itself?

Allow me to be a contrarian and say that feature selection is overrated. My post here discusses feature selection when features are correlated, but the same bias-variance argument applies to uncorrelated features. Briefly, by excluding features, you are forcing their coefficients to be zero, and even if this results in lower variance, you might incur enough of a bias (by forcing the estimate of a nonzero parameter to be zero) that you are worse-off for having done the feature selection.
Further, feature selection tends to be unstable. Our Frank Harrell, founding chair of the Department of Biostatistics at Vanderbilt University, has written about this. (That Regression Modeling Strategies book he mentions should be considered mandatory reading for anyone doing predictive modeling (which includes all supervised machine learning).) If you want to get into specific relationships, particularly causal relationships, that requires much more delicate handling than predictive modeling requires. As Harrell mentions in the link, selecting a feature is related to overestimating its effect; conversely, electing to omit a feature is related to underestiating its effect.
Note that your desire to do feature selection is driven by your use of an improper scoring rule. You are correct that your class imbalance is likely driving you not to predict the minority class, but this is probably correct behavior. The minority class might always be unlikely. By using improper scoring rules, you are missing this fact and missing out on being able to predict the probability of being in your minority class. Classification requires a threshold, which your software default is likely to have set at $0.5$, even though the minority class might never truly have a probability of $0.5$ under any conditions (combination of features). The simplest way to deal with this is to adjust the threshold. A more advanced approach would deal directly with predicted probabilities. Again, Frank Harrell has written about this in two great blog posts [1, 2].
Finally, with only six cases of the minority condition, it is unlikely that you have enough data to do much at all, even if you use best practices, and it is important to be able to relay to stakeholders when they are pursuing a hopeless problem or using a hopeless approach.
