Hi @mrshenli,
Thanks for your reply. My torch version is 1.6.0 and I receive no warnings/errors.
I’ve tried:
for _ in range(3):
    output = ddp_model(torch.rand(20, 10)
    loss_fn.backward(output, labels)

optimizer.step()

This works as I expected.
When you said fw([1, 2, 3]) -> bw([1, 2, 3]), do you mean the following?
outputs = (ddp_model(torch.rand(20, 10)), ddp_model(torch.rand(20, 10)), ddp_model(torch.rand(20, 10)))

for i in range(3):
     (loss_fn(outputs[0], labels).backward(retain_graph=True),  loss_fn(outputs[1], labels).backward(retain_graph=True), loss_fn(outputs[2], labels).backward(retain_graph=True))

optimizer.step()

This still doesn’t synchronize the weights and doesn’t throw any error.