You can try to had regularization like L2 by setting weight_decay to your optimizer like:
optimizer = torch.optim.Adam(model.parameters(), weight_decay= 0.2)
I think that for the L1 you have to implement it “manually”.
You can also try to put some weights to your class proportionally to their distribution. Which kind of loss do you use ?
Have you tried to balance the class ratio between your training and validation test?