Hi all, I have encountered a few strange problems when I get started to use DDP in PyTorch.
Exp. setting
I used 2 NVIDIA GPUs in ONE machine as well as 14 threads on 4 CPUs.
I spawned 2 processes each for a GPU in a program.
Each program just allocates 1+G GMEM.
Questions for help


Though I have noticed to use map_location to load state_dict and so on, I found it is good for GMEM to be even across the GPUs and the somewhat strange thing was there were occasionally large fluctuations in GPU utilization, like one is 90+%, the other is ~20%, or the reverse reported. I thought DDP could lead to a more balanced load… Maybe it was because the CPUs were still in need so that the GPUs were far from busy, right?


When I was trying to utilize the CPUs, I ran 3 programs on these 2 GPUs. However, it was found to get stuck at a time (in the middle process of training). It appeared to be all 3 programs reduced one of their spawned processes and only left the other on the same GPU. The CPUs were not in use but the GPU utilization shown by nvidia-smi was 100% though it was obvious that the processes were stuck. I am wondering was it the case of a deadlock in that 2 programs were the best practice in the case? (might be somewhat general)


(suspect the stuck is caused by?
val_loss, val_acc = self._run_epoch(...)

dist.barrier()

val_acc = torch.tensor(val_acc).unsqueeze(0).cuda()
out_val_acc = [torch.zeros_like(val_acc) for _ in range(dist.get_world_size())]
dist.all_gather(out_val_acc, val_acc)
val_acc = torch.cat(out_val_acc).mean().item()  # based on the exact division

Thanks & Regards