If you are using two different DataLoaders (one with the standard sampler, the other with the WeightedRandomSampler), you could also pass different transformations to the datasets and thus make sure the augmentations are only applied to one of them.