Using P-value tests for feature selection in machine learning? Say I have a dataset $X$ consisting of differing RVs which may be continuous and/or nominal and my target set which again may be continuous or nominal. If the number of covariates is large, I would typically want to boil those down into features which hold the greatest importance in my prediction. As such, I would normally conduct hypothesis tests to understand this, however I am under the impression that such tests can be misleading as to whether to include or reject the null hypothesis, and thus drop or include the mentioned columns.
For example, in my ANOVA test, I assume as my null hypothesis all groups of some nominal RV have equal variance and equal mean, i.e. the nominal RV doesn't distinctly categorise between groups. I proceed to conduct my hypothesis test and it suggests strongly, with a p-value of $0.001$, that the group have an equal variance and equal mean, and thus, since it doesn't distinctly categorise each group, that the RV should be dropped. Why in such case would it be a bad idea to not drop this RV?
Any help is appreciated.


If the number of covariates is large, I would typically want to boil those down into features which hold the greatest importance in my prediction.

Why do you think you would want to do this if you were not constrained in some other way (e.g. cost of acquisition or memory to store the features)?
Anyway, I've answered a similar question here.  The main point is that a small p value gives you absolutely no information about the size of the effect, which is the main consideration.  Small and negligible effects can be highly significant.  As per my example in the linked answer, the variable Z would be included in the model based solely on significance criteria, yet the model performance is nearly identical with out without it meaning selection using p values can lead you to select unimportant variables.
