Ah, ok, sorry, I didn’t notice the part about using the multiprocessing package separately. I think this is because I had the same issue just using DDP and Dataloaders with num_workers > 0. I get these errors just before my script exits; I already am a good citizen by using dist.barrier() to wait for all training processes to complete before exiting.
When I set the num_workers to zero the script exits smoothly without any errors. In the past I have often used multiple DataLoader workers in a DDP context and never had this issue. I’m wondering if this is something that sneaked in to a recent Pytorch release or is related to recent a CUDA version?
My setup:
| NVIDIA-SMI 470.86 Driver Version: 470.86 CUDA Version: 11.4     |

$ pip show torch
Name: torch
Version: 1.10.0

This is on a 4x V100 machine with Ubuntu.
It might also be in my script of course.