Interesting to find this 2-year-old post; I just ran into the same problem.  My guess about the reason it’s forbidden is that when you are doing distributed processing, especially with an IterableDataset and/or a DistributedSampler which passes a different part of a dataset to each process, Pytorch can lose track of what has already been assigned if the user is allowed to modify the dataloader’s attributes on the fly.