Should the same cross-validation method be used across multiple models? The assignment is to write a simple ML program that trains and predicts on a dataset of our choice. I want to determine the best model for my data. The response is a class (0/1). I wrote code to try different cross-validation methods (validation set, leave-one-out, and k-fold) on multiple models (linear regression, logistic regression, k-nearest neighbors, linear discriminant analysis). Per model, I report the MSE for each cross-validation method and track the lowest one. I then pick the model with the lowest tracked MSE. This is where I think I went wrong. If I am cross-validating multiple models, should I use the same cross-validation method?

What is the reason that you have different models? What are you doing with the models?

I want to determine the best model for my data.

For determining the best model it doesn't matter whether you did cross validation differently. What is important is that you compare the models with an independent test dataset.
