maralm:

No, I am trying to just run a forward layer and compute backward with autograd.backward on that layer instead of running loss.backward().


I see. DDP does not work for this case yet. Currently, all outputs you get from DistributedDataPararlel.forward() must participate in the same backward pass, otherwise, it would mess up with DDP’s internal communication state. Hope this can help explain that: https://pytorch.org/docs/master/notes/ddp.html#internal-design 89