Sorry, I misunderstood the question. If I understand it now correctly, you would like to only augment data samples, which were re-drawn while the first occurrence of the sample should not be transformed?
If that’s the case, I think the cleanest approach would be to write a custom sampler, reuse the WeightedSampler logic and add another flag to indicate if the current sample index should be transformed or not.
You could theoretically also track the indices in the Dataset, but it would break if you are using multiple workers (which is the common case).