The docs 7 explain this behavior and suggest to use the worker information:

When a subclass is used with DataLoader 3, each item in the dataset will be yielded from the DataLoader 3 iterator. When num_workers > 0, each worker process will have a different copy of the dataset object, so it is often desired to configure each copy independently to avoid having duplicate data returned from the workers. get_worker_info() 1, when called in a worker process, returns information about the worker. It can be used in either the dataset’s __iter__() method or the DataLoader 3 ‘s worker_init_fn option to modify each copy’s behavior.