I think you might pass the wrong weights to WeightedRandomSampler.
The sequence of weights should correspond to your samples in the dataset.
Here is a small example:
weights = 1. / torch.tensor(class_sample_counts, dtype=torch.float)
samples_weights = weights[train_targets]

sampler = WeightedRandomSampler(
    weights=samples_weights,
    num_samples=len(samples_weights),
    replacement=True)