Solved by ptrblck in post #2 
              
              
                The loss calculation is performed of each node and doesn’t need any communication. In case your loss needs to communicate some gradients, I guess it could be used as part of the model (and would thus be treated as any nn.Module).