Can I compare different hyperparameters combination performance on testing dataset I am runing a GridSearch with different range of hyperparameter values in order to find the best ones based on performance metrics (F1, AUC, etc.). I however have an imbalanced dataset, so I need to undersample my training dataset to have a balanced amount of binary (0 or 1) event and non-event samples for the training.
For this reason, I am wondering if it is OK to train multiple models on the training dataset (with different hyperparameter values) and compare their performance on the testing dataset to chose the best hyperparameter values.
I know the "normal" method is to chose hyperparameters on the validation dataset (with cross-validation on a portion of the training dataset), but I feel like it would make more sense to chose them based on their performance on the testing dataset, considering it is not balanced (just like the future real world data that will be used as inputs in the model).
According to my reflexion, it would be OK because I would still train my model on the training dataset and test it on unseen data (testing dataset).But I would like to confirm that it does make sense?

The short answer is no; if you choose an option (hyperparameters) based on results for a particular data set then that data set should be regarded as the test set.  Any decisions made should be regarded as part of the training process and so need to be verified on a further data set.
An exception might be where the data set is believed to be large enough or similar enough to the target data.  Whether you can justify that depends on the application and level of security.
Another exception might be where all your models performed acceptably well on the test data set, according to some acceptance criteria.  In that case, although you are choosing one for deployment, you have confidence that the decision doesn't affect the acceptability of the solution.
