Hi
Can you give more details about how you are training on multiple GPUs.
For training on MultiGPUs, One way is to use DataParallel() where batches of input data are split across GPUs and after each step of computation gradient accumulation happens on a single GPU. You can refer to this tutorial link 20.