Must we do feature selection in cross validation? It is me again, asking very similar questions with reference to this post, this time, I am a bit confused whether one should perform feature selection before KFold cross validation. I often times see people do one-shot feature selection prior to modelling, and wonder if this is normal?
In one of the top comment, he mentioned:

I don't think that is (quite) what Hastie, et al. are advocating. The general argument is that if feature selection uses the response then it better be included as part of your CV procedure. If you do predictor screening, e.g., by looking at their sample variances and excluding the predictors with small variation, that is ok as a one-shot procedure.


Wonder if that means we can do one shot feature selection before cv if we do not take response variable into account?
In fact, if I were to do the feature selection within cross validation, then is there any use of doing extensive EDA on the feature selection?

What if there is many multi-collinearity features in my dataset, should I handle it before cross-validation?

Edit:  I have been reading up extensively on this issue, but it is really cracking my head because of the uncertainty of when it is "ok" to do feature selection/reduction prior to CV. I chanced upon this very short piece of code from scikit-learn in which they used Ward's Method to remove highly correlated features by choosing a cut off point in the clustering process. I wonder if this is something that I can do outside the loop, and if yes, how does one choose the "threshold"?

In principle, if you want your CV validation scores to reflect what applying an algorithm trained in the manner you did will do on new data, then everything should be part of the cross-validation (or bootstrapping, or whatever else you do along those lines). It's of course the most concerning when we need to make decision on this basis (e.g. Which of several approaches that might be affected differently by deviations from the ideal should I pick?). This ideal is not entirely achievable in practice (except for when you reserve a single separate validation and/or test partition, which is however inefficient for anything but huge datasets) and exploratory data analysis is an obvious example of that.
Some violations of this ideal are worse than others. E.g. you may have to do some EDA to define what your cross-validation scheme should look like (for example, you may discover that you have multiple records per patient and maybe for that reason you should use group-K-fold instead of basic K-fold). Similarly, screening predictors for whether they have zero (or near-zero) variance seems pretty harmless. Creating features solely based on human understanding of the task (e.g. grouping together different mis-spellings of a category name) is usually also totally unproblematic.
Where you are definitely crossing the line into extreme danger is when you do target encoding (representing categories by their mean outcome), that must always be done within the cross-validation loop (or even within an additional CV loop within that). Really, most things that use the prediction target should be considered too dangerous to be done outside the CV loop.
Many other things may be a bit between these extremes. E.g. transforming predictors (e.g. standardization or doing PCA) or imputation of missing predictors ideally belongs in the CV loop, too, but it is less immediately obvious how much (clearly at least a little bit) this would undermine the validity of the CV evaluation. The more you deviate though, the more a final external validation on new data becomes more important (it may of course also be very important for other reasons such as a mismatch of where the training data comes from vs. where the model would be used).
