One reason might be that CUDA GPU shows 100% utilization when running NCCL collective communications, even if it is actually block waiting for other peers to join and doing nothing. So the GPU utilization number cannot faithfully represent how busy a GPU is.