Thanks for the quick answer. WeightedRandomSampler sounds promising, however I plan on changing the set of used samples quite often.
So I think it makes more sense to set some kind of boolean tensor within the dataloader and then modify the dataloader instead of the sampler.