That does not solve the randomness problem inside numpy+pytorch. For example, you will get exactly the same numpy random numbers in each epoch! Does anyone know a “clean” solution to this? I have to  call (from inside dataset.getitem) a very messy (possibly wrong) code to make every worker have different numpy random seed at different epochs.
def set_cuda_rand_seed():
worker = torch.utils.data.get_worker_info()
new_seed = np.random.randint(0, 2 ** 32 - 1)
if worker is not None:
new_seed = worker.seed
new_seed = int(new_seed) % (2 ** 32 - 1)
new_seed = int(new_seed)
random.seed(new_seed)
np.random.seed(new_seed)  # todo <— bug using num_workers>0
return new_seed