Hello,
I am having the problem that as I scale to multiple GPU’s my training is bottlenecked at the capacity of a single GPU. So for example, I am using 8 GPU’s with 16GB of GPU memory per GPU. However, GPU 0 uses approximately it’s full capacity but, the other GPU’s only use 11GB’s. I know that one GPU is designated as the main GPU and requres memory to coordinate but, is there any way to make this more efficient? A printout of my nvidia-smi usage can be found below. Thank you.

GPU_Utilization.png654×619 55.7 KB