@wanchaol Pytorch automatically splits my batch across the 8 GPUs and compute 8 Losses with 8 backpropagations. Is there a way to prevent this and compute only 1 Loss with 1 backpropagation?
eg. if my batch is n=80 pytorch will split n/8=10 sub batch on each GPU and compute 8 Losses, unfortunately a loss computed on 10 datapoint is less consistent than a loss computed on 80 datapoint, that’s why I’d like to avoid this to happen.