Must we do feature selection in cross validation? It is me again, asking very similar questions with reference to this post, this time, I am a bit confused whether one should perform feature selection before KFold cross validation. I often times see people do one-shot feature selection prior to modelling, and wonder if this is normal?
In one of the top comment, he mentioned:

I don't think that is (quite) what Hastie, et al. are advocating. The general argument is that if feature selection uses the response then it better be included as part of your CV procedure. If you do predictor screening, e.g., by looking at their sample variances and excluding the predictors with small variation, that is ok as a one-shot procedure.


Wonder if that means we can do one shot feature selection before cv if we do not take response variable into account?
In fact, if I were to do the feature selection within cross validation, then is there any use of doing extensive EDA on the feature selection?

What if there is many multi-collinearity features in my dataset, should I handle it before cross-validation?

Edit:  I have been reading up extensively on this issue, but it is really cracking my head because of the uncertainty of when it is "ok" to do feature selection/reduction prior to CV. I chanced upon this very short piece of code from scikit-learn in which they used Ward's Method to remove highly correlated features by choosing a cut off point in the clustering process. I wonder if this is something that I can do outside the loop, and if yes, how does one choose the "threshold"?

Cross-validation is a means of estimating the performance of a method for fitting a model, rather than of the model itself, so all steps in fitting the model (including feature selection and optimising the hyper-parameters) need to be performed independently in each fold of the cross-validation procedure.  If you don't do this, then you will end up with an optimistically biased performance estimate.  See my (with Mrs Marsupial) paper on this topic
GC Cawley and NLC Talbot, "On over-fitting in model selection and subsequent selection bias in performance evaluation", The Journal of Machine Learning Research 11, 2079-2107 (pdf)
I tend to use nested cross-validation to get an unbiased performance estimate, but if you don't need an unbiased performance estimate, just choose between competing methods (that don't have too many degrees of freedom, i.e. not feature selection!) then that often isn't necessary in practice, see Wainer and Cawley
J Wainer and G Cawley, "Nested cross-validation when selecting classifiers is overzealous for most practical applications", Expert Systems with Applications 182, 115 (doi:10.1016/j.eswa.2021.115222)
Once you have that performance estimate, then retrain the model on the whole dataset, repeating the feature and model selection procedures once more.
Also I would advise against feature selection if the aim is to improve performance (rather than identifying relevant features itself being the goal).  Using a regularised model will often perform better.
