I am currently using DataParallel. I am using batches yes. What iam thinking about is i saw someone normalizing the loss of different GPUs, so i did similar thing , and just loss.backward the normalized loss of the 8 GPUs, shouldn’t this propagate on the default GPU computation grpah ?