@ptrblck Thanks for your reply.

Yes i am talking about DistributedDataParallel, but the strange thing is that it happens even without DistributedDataParallel i.e., using a single GPU. This means that the problem doesn’t come from NCCL calls (correct me if i am wrong).
In case of single GPU after some 30000 to 40000 samples being processed, the training gets stuck, and the GPU utilization goes to 0% from 95%, whereas the CPU utilization fluctuates between minimum and 100%.
The temperature of the GPU remains around 70C at 95% usage (at smooth training), which is normal, i think, and is around 55C at 0% usage (when the problem occurs).
When this problem occurs, it also effects other processes on the machine. For example, i am processing the VoxCeleb dataset, which slows down to minimum when this problem occurs.
Also, i have tried with different versions of PyTorch, i.e., 1.6.0, 1.7.0, 1.8.1, and the latest 1.9.0. All behave the same.
Also, i have checked on different machines, with different GPUs (Machine 1 with Tesla T4, and Machine 2 with Titan RTX), i faced the same problem.
By the way i am trying to train this model, where other people do not have this problem. I have no idea why it happens in my case?