Must we do feature selection in cross validation? It is me again, asking very similar questions with reference to this post, this time, I am a bit confused whether one should perform feature selection before KFold cross validation. I often times see people do one-shot feature selection prior to modelling, and wonder if this is normal?
In one of the top comment, he mentioned:

I don't think that is (quite) what Hastie, et al. are advocating. The general argument is that if feature selection uses the response then it better be included as part of your CV procedure. If you do predictor screening, e.g., by looking at their sample variances and excluding the predictors with small variation, that is ok as a one-shot procedure.


Wonder if that means we can do one shot feature selection before cv if we do not take response variable into account?
In fact, if I were to do the feature selection within cross validation, then is there any use of doing extensive EDA on the feature selection?

What if there is many multi-collinearity features in my dataset, should I handle it before cross-validation?

Edit:  I have been reading up extensively on this issue, but it is really cracking my head because of the uncertainty of when it is "ok" to do feature selection/reduction prior to CV. I chanced upon this very short piece of code from scikit-learn in which they used Ward's Method to remove highly correlated features by choosing a cut off point in the clustering process. I wonder if this is something that I can do outside the loop, and if yes, how does one choose the "threshold"?

One of the most challenging elements of this problem is knowing when it's OK to put unsupervised learning steps outside of the CV loop and when they should be fully penalized for by including them inside the loop.  Generally speaking, unsupervised learning procedures such as principal components analysis can be unstable, i.e., the loadings of the first principal component will change when computed on a new sample.  And unsupervised learning steps to exclude features, such as redundancy analysis and variable clustering, can also be unstable.  But their instabilities can either hurt you or help you, i.e., may raise or lower your final $R^2$.  So they don't consistently work in your favor.  Overfitting in a final predictive discrimination measure such as $R^2$ or pseudo $R^2$ comes from doing things that consistently "work" in your favor, such as doing feature selection using supervised learning, whether manual or automated.
So it's generally OK to keep completely unsupervised learning steps outside the CV loop as pre-processing steps, but sometimes bring them into the loop to make sure your final performance measure doesn't suffer from instability in unsupervised learning.
