Does feature selection always benefit the performance of a ML model? In my ML pipeline, I normally perform feature selection, by performing a few of the tests mentioned below, the ones relevant to the model. I tend to drop features with negative outlier to the rest of the features, but I wonder whether more data is better, and leave the feature selection to the algorithm to dechiper which features to prioritise.

*

*Is identifying and removing extremely multicollinear regression variables always in benefit of the performance of such a model?

*Is identifying features in a classification model which do no depend on the label set via a Chi-squares test for independence always in benefit of the performance of such a model?

*The features_importances function in sklearn measures the average gain of purity by splits of a given variable. In the cases where the importance level is extremely low, it is always advisable to drop such features.


Feature selection often makes generalisation performance worse rather than better, so I would recommend against routinely including it in the analysis unless identifying the relevant attributes is a specific aim.
Regularisation (which is used in most modern ML algorithms) is able to deal with colinearity.
Testing individual features for independence is not a good idea as some features may be useless on their own, but highly informative in combination with others.  Consider a 2-d dataset where one class consists of a spherical blob at the origin and the other class consists of a doughnut centered on the origin.  In that case, you may be able to get perfect separation with both features, but a chi-squared test will probably reject both features considered individually.
