pritamdamania87:




 Glory_Chen:


Though I have noticed to use map_location to load state_dict and so on, I found it is good for GMEM to be even across the GPUs and the somewhat strange thing was there were occasionally large fluctuations in GPU utilization, like one is 90+%, the other is ~20%, or the reverse reported. I thought DDP could lead to a more balanced load… Maybe it was because the CPUs were still in need so that the GPUs were far from busy, right?



What are the CPUs used for in your program? It could be possible that if CPUs were busy doing something and not pushing compute to GPUs, the GPU utilization might drop.


I used taskset command to specify the 14 threads on the CPUs to use.






 pritamdamania87:




 Glory_Chen:


When I was trying to utilize the CPUs, I ran 3 programs on these 2 GPUs. However, it was found to get stuck at a time (in the middle process of training). It appeared to be all 3 programs reduced one of their spawned processes and only left the other on the same GPU. The CPUs were not in use but the GPU utilization shown by nvidia-smi was 100% though it was obvious that the processes were stuck. I am wondering was it the case of a deadlock in that 2 programs were the best practice in the case? (might be somewhat general)



I’m not sure I followed this completely, could you elaborate? What are these 3 programs running? Are you running DDP in 3 processes but across 2 GPUs? In general if GPUs are stuck at 100% util it probably indicates an issue where a NCCL collective op is stuck.


Every program uses mp.spawn to spawn 2 processes on the 2 GPUs. I ran 3 parallel programs. The command can be thought of,
CUDA_VISIBLE_DEVICES=3,5 taskset -c 21-27,35-41 python xxx.py --world_size 2 --port 12355
CUDA_VISIBLE_DEVICES=3,5 taskset -c 21-27,35-41 python xxx.py --world_size 2 --port 12356
CUDA_VISIBLE_DEVICES=3,5 taskset -c 21-27,35-41 python xxx.py --world_size 2 --port 12357

Thanks.