Have you tried to increase the weight_decay ?
If you are using CrossEntropyLoss, you can put weights to your classes like :
weights = torch.one(nb_classes)

weights[0] = 2 #if you want to set the weight of class 1 to 2 etc...

criterion = nn.CrossEntropyLoss(weight=weights)

Have you set Dropout module ?