Solved by dazzle-me in post #4 
              
              
                Well, I think you can maintain some kind of “don’t load this image anymore” list, which will contain img’s names on which your cross_entropy  < alpha after training. 
Consider an example : 
class RestrictedDataset(torch.utils.data.Dataset):
    def __init__(self, img_directory, banned_images):
     …