Should the same cross-validation method be used across multiple models? The assignment is to write a simple ML program that trains and predicts on a dataset of our choice. I want to determine the best model for my data. The response is a class (0/1). I wrote code to try different cross-validation methods (validation set, leave-one-out, and k-fold) on multiple models (linear regression, logistic regression, k-nearest neighbors, linear discriminant analysis). Per model, I report the MSE for each cross-validation method and track the lowest one. I then pick the model with the lowest tracked MSE. This is where I think I went wrong. If I am cross-validating multiple models, should I use the same cross-validation method?

As you noted, using various cross-validation techniques can produce various results for various models, making it challenging to compare the effectiveness of various models in an objective way. As a result, using the same cross-validation method across multiple models is recommended. In this way, you can make fair comparisons between the various models' outputs.
Also, if your objective is to find the best model, is this strategy effective? According to machine learning's "no-free-lunch theorem," no single model outperforms all others in all circumstances. The best model for your data will be determined by the structure and size of your data, the problem's complexity, the model's parameters, and the computational resources available.
As a result, it is essential to choose the model and cross-validation method that are "the best" for your data and objectives. You could also consider techniques like hyperparameter optimization.
I hope this helps!
