I see. In that case, DDP alone won’t be sufficient, as DDP’s output and loss are local to each process. If you only need to calculate the globally loss, one option is to gather the outputs instead of loss, and then calculated loss on the gathered outputs. If you also need back propagation from the global loss, there are at least two options:

combine DDP with RPC: Use DDP to compute local output and use RPC to collect them into one process and compute global loss, and then use distributed autograd to start the backward pass from global loss. This tutorial 10 shows how to combine DDP with RPC.
Use the autograd-enabled collective communications in torch.distributed.nn.functional 1. Tests in this PR 4 can serve as examples. (This feature is not officially released yet.)