I recommend using a balanced dataset for evaluation because on an imbalanced dataset, e.g., accuracy is misleading. The loss function would only be applied during training then (I assume you want to measure sth like “accuracy” on the validation/test set).
Furthermore, to get more insights into the classification performance with and without loss weighting during training, i recommend looking at the confusion matrices on the validation/test sets when you train with and without weighting.



 apganesh:

Do we apply the class weights to the loss function for validation/dev set?
If so, would it not mislead us from the actual target?


It depends on what you actual target is – that’s something you need to decide depending on the goal for which you train the model.
Lastly, a metric that might be helpful here would be the “average-per-class accuracy” – I’ve summarized it here: http://rasbt.github.io/mlxtend/user_guide/evaluate/scoring/ 14


Screen Shot 2019-05-27 at 2.31.35 PM.png640×508 33.5 KB


I’m not sure if e.g. early stopping should also be used with a weighted validation loss, so maybe

I’d say the concept of early stopping is not necessarily related to the class imbalanced problem. It can be used if you notice an increasing degree of overfitting (measured on the validation set) when you continue training, but this can happen with/without class imbalance. I don’t think there’s a direct relationship between these two issues.