Used DistributedDataParallel, but find that the speed with 4 nodes (4 GPUs per node) is sometimes even slower. Sometimes, the job runs 4300 images for each second, which is normal. If it is 4300 at the beginning, the job will always be running at this fast speed. But sometimes, the job is running at 1000 images per second, and the whole job will be at this speed always. The jobs are running in a cluster, and in different physical machines but same machine type.
For the job with problematic issues, the GPU utility is always 98%~100%. Pytorch version = 1.4; CUDA=10.1; Ubuntu 16.04 docker image. NCCL is for sure to use the Infinity band with the following logs. The time cost of data loading is also very small (less than 1%).
Are there any idea to debug?
41de615398b349e78486287e94d4883b000000:1573:1573 [0] NCCL INFO NET/IB : Using [0]mlx4_0:1/IB ; OOB eth0:10.0.0.4<0>