Thank you for your reply.
This is exactly what i want to do. Like i have batch size of 1024 and with 8 gpu , each gpu has a batch size of 128. However during the loss.backward() , the backpropagation calculate as if the batch size was 1024 not 128 so it runs out of memory. What i really want is to write code that calculate the loss over the 8 gpus, then sum it up for the optimizer to run. Any guidance on where to find that guide