Hi, I also fell into the same problem you are facing.
I tried oversampling of minority classes balancing them to the majority one but that didn’t give me good results.
Some things I would suggest you try also from @tom’s suggestion:

When you are using pre-trained model and fine tunning the final FC layer or a couple of final layers they tend to overfit. Dropout layer with 20% drop can help the model to generalize.
Try weighted CrossEntropyLoss which worked amazing in my case. For weight assembling I followed the theory below:

nSamples = [887, 6130, 480, 317, 972, 101, 128] #of class instances
normedWeights = [1 - (x / sum(nSamples)) for x in nSamples]
normedWeights = torch.FloatTensor(normedWeights).to(device)
Criterion = nn.CrossEntropyLoss(weight=normedWeights)


Normalize the data with mean and standard deviation.
Use a lot of data augmentation like flips, rotation, color plane, jitter, and others.
Another thing you can do is make sure there is an equal ratio of instances in a single minibatch going to the network. You can do that customizing Dataset.

Hope it helps.