I think you need to set the seed in the worker_init_fn as described in the docs 53:

By default, each worker will have its PyTorch seed set to base_seed + worker_id , where base_seed is a long generated by main process using its RNG. However, seeds for other libraies may be duplicated upon initializing workers (w.g., NumPy), causing each worker to return identical random numbers. (See My data loader workers return identical random numbers 36 section in FAQ.) You may use  torch.initial_seed()  29 to access the PyTorch seed for each worker in worker_init_fn , and use it to set other seeds before data loading.