If could provide a WeightedRandomSampler with all weights set to zero for elements of the unwanted classes.
Alternatively, you could write your custom Dataset and just read all images from a particular folder, which will also result in a single class being sampled.