One other thing I noticed is that either num_workers and nproc_per_node must be the same or num_workers must set 0 for the code to run. Do you know the reason behind this?
Example in point: the code runs with num_workers=8 and nproc_per_node=8 and num_workers=0 and nproc_per_node=8 but not with num_workers=10/9/7/6 (any of these) and nproc_per_node=8 (DataLoader worker (pid(s) 53785) exited unexpectedly.).
I’m using persistent_workers=True when running with num_workers>0 and persistent_workers=False with num_workers=0.