Are you trying to limit the number of unique images that are used? You can probably do this by editing the DatasetFolder dataloader:
https://pytorch.org/vision/stable/_modules/torchvision/datasets/folder.html#datasetfolder
or by simply creating a modified dataset on disk with some of the images removed.