Even though your class distribution is not very imbalanced, it seems your model is still overfitting to this class.
You could e.g. add the weight parameter to your criterion and try to counter this effect.