Hey @albert.cwkuo
With the above code, I think DDP still syncs all grads for both loss1 and loss2, because the flag controlled by no_sync ctx manager is used when calling DistributedDataParallel.forward(). So, as the forward is out of the no_sync context, DDP would still prepare to sync all grads during the backward pass.

  
      github.com
  
  
    pytorch/pytorch/blob/5036c94a6e868963e0354fc04c92e204d8d77677/torch/nn/parallel/distributed.py#L477-L498 8

@contextmanager
def no_sync(self):
    r"""
    A context manager to disable gradient synchronizations across DDP
    processes. Within this context, gradients will be accumulated on module
    variables, which will later be synchronized in the first
    forward-backward pass exiting the context.

    Example::

        >>> ddp = torch.nn.DistributedDataParallel(model, pg)
        >>> with ddp.no_sync():
        ...   for input in inputs:
        ...     ddp(input).backward()  # no synchronization, accumulate grads
        ... ddp(another_input).backward()  # synchronize grads
    """
    old_require_backward_grad_sync = self.require_backward_grad_sync
    self.require_backward_grad_sync = False
    try:
        yield


  This file has been truncated. show original

  
  
    
    
  
  


How is MyModel implemented. Does it contain two independent submodules sub_model1 and sub_model2 and do sth like the following?
class MyModel(nn.Module):
    def __init__(self):
        self.sub_model1 = SomeModel1()
        self.sub_model2 = SomeModel2()

    def forward(self, input):
        return self.sub_model1(input), self.sub_model2(input)