Hi,
I tried to run 2 training jobs on a GPU cluster with 8 GPUs, both jobs need to use NCCL AllReduce. And I notice the training speed is slower if running the 2 jobs at same time than running them separately. Is this because they are competing for the bandwidth between GPU communications (the AllReduce call)? Thanks.