Ok so it seems that you have same ratio on both sets and that the classes have the same weights (because they have equal number of images in total).
Are you doing a cross validation on your training dataset during training ?
Have you tried to increase the weight decay ?