Well, I think you can maintain some kind of “don’t load this image anymore” list, which will contain img’s names on which your cross_entropy  < alpha after training.
Consider an example :
class RestrictedDataset(torch.utils.data.Dataset):
    def __init__(self, img_directory, banned_images):
        self.img_pathes = [image_name for image_name in os.listdir(img_directory) \
                           if not image_name in banned_images]
    def __getitem__(self, index):
        image_name = self.img_pathes[index]
        # somehow load image
        # image = ...
        return image, image_name
    def __len__(self):
        return len(self.img_pathes)

And then during training you will have to discard image_name (because there no use of it during training)
But when when the training is done, and you want to remove some images that pass the threshold, - you’ll have this image_name to add it to banned_images list, hope you get the idea.
P.S. you will have to recreate dataset and dataloader after each training cycle, but I think it won’t bring any overhead in your solution