Did neither the WeightedRandomSampler nor the loss function weighting help at all?
Is your data sorted in any way? If so, could you use class weighting for your criterion and a SubsetRandomSampler?
Also, could you post the code for using the WeightedRandomSampler so that we can make sure it was used properly?