I realized it was an issue with the way I deleted items, should be:
   data = np.delete(self.data, remove_list, axis=0)
    
        
        targets = np.delete(self.targets, remove_list, axis=0)

But is it doing the correct thing overall?:  Removing specific images based on the index or is the index different every time it is loaded?