Be careful with trying to manipulate the Dataset in a DataLoader, if you are using multiple workers.
Each worker will use a clone of the Dataset, so that changes to the internal states of the Dataset will not be reflected.
Could this be the case for your issue?