Hyperparameter tuning on the training data? Cross validation I have a set of data (around 1500 data points) with 75 parameters and I am trying to compare the performance of SVM, Decision Tree and a few other supervised techniques.
My data set is not perfectly balanced (900 - Cat 1 vs 600 - Cat 2), so I decided to run SMOTE method to balance it. Here is what I did.

*

*I split the data 80% to 20%

*Used SMOTE on the training data

*Then, used grid-search to optimize my parameters on the training data

*Then, tested & compared the models using the testing data

I was told that due to my data size, I should have used cross validation method. Also, I was told that hyperparameter optimizations should have been done with a validation division.
Here is what I think but I may be wrong (I decided not to use SMOTE anymore since the date set is not terribly imbalanced.

*

*Use grid-search to optimize hyperparameters.

*Use cross-validation to generate results

*Then, compare methods (AUC, F1 e.g.)

I have a feeling that this is not right. Should I have a validation fold/file and tune hyperparameters there?

What you were told is correct, i.e.:

*

*If you tune parameters, you have to do an inner / outer split, tune on the inner and evaluate the performance of the final model on the outer split

*If you have the computing power, CV is less wasteful than a simple validation. However, note that with an inner / outer CV, computing time goes as k1*k2, where k1 is the number inner and k2 the number of outer folds, so in many cases people choose at least for the outer split a simple validation. However, in principle, using CV for both splits will make best use of your limited data.

