@jpeg729 No worries. I get your point: the three parallel networks ought to have the same same gradients to train identically, but I think PyTorch’s DataParallel module automatically sums the gradients before backprop. I will check that out.
All of this being said, this is exchange has tangential to my original question.
Anyone know why torch.mean(A) over when A is a BATCH_SIZE x 1 variable with requires_grad=True doesn’t seem to stay reduced on backward()?