Must we do feature selection in cross validation? It is me again, asking very similar questions with reference to this post, this time, I am a bit confused whether one should perform feature selection before KFold cross validation. I often times see people do one-shot feature selection prior to modelling, and wonder if this is normal?
In one of the top comment, he mentioned:

I don't think that is (quite) what Hastie, et al. are advocating. The general argument is that if feature selection uses the response then it better be included as part of your CV procedure. If you do predictor screening, e.g., by looking at their sample variances and excluding the predictors with small variation, that is ok as a one-shot procedure.


Wonder if that means we can do one shot feature selection before cv if we do not take response variable into account?
In fact, if I were to do the feature selection within cross validation, then is there any use of doing extensive EDA on the feature selection?

What if there is many multi-collinearity features in my dataset, should I handle it before cross-validation?

Edit:  I have been reading up extensively on this issue, but it is really cracking my head because of the uncertainty of when it is "ok" to do feature selection/reduction prior to CV. I chanced upon this very short piece of code from scikit-learn in which they used Ward's Method to remove highly correlated features by choosing a cut off point in the clustering process. I wonder if this is something that I can do outside the loop, and if yes, how does one choose the "threshold"?

The thread you mentioned already discusses it in great detail, so I'd skip the parts that were already mentioned there. Answering your question, it depends on what you mean by "one shot feature selection before cv if we do not take response variable into account". For example, if you look at the data and discover that some of the features are very low quality (the data doesn't make sense, it is obviously wrong, or the feature is constant) than yes, you can do this outside of cross-validation. On another hand, doing in-depth exploratory data analysis to pick the features by hand has the same effect as doing similar things algorithmically and there is no reason why "by hand" you won't produce an overfitting model.
