My Train and Validation set comes from the exact same imbalanced class distribution. I am using weighted CrossEntropy loss function to calculate training loss. As it would be unfair to use weighted loss function for Validation, I interpret from above discussion, not to calculate validation loss and rely on Validation accuracy only. What criterion should I use for early stopping in this case? Or the plain CrossEntropy loss without weights can be reasonable for validation. Please let me know your suggestion. Thank you.