Hey @lkp411
no_sync() disables the following code, which should completely disable communications across processes (unless you are using DDP.join() 11).

  
      github.com
  
  
    pytorch/pytorch/blob/b1907f5ebcaeb7405aa3d9d8025a7a5eeb0ee590/torch/nn/parallel/distributed.py#L710-L720 17

if torch.is_grad_enabled() and self.require_backward_grad_sync:    self.require_forward_param_sync = True    # We'll return the output object verbatim since it is a freeform    # object. We need to find any tensors in this object, though,    # because we need to figure out which parameters were used during    # this forward pass, to ensure we short circuit reduction for any    # unused parameters. Only if `find_unused_parameters` is set.    if self.find_unused_parameters:        self.reducer.prepare_for_backward(list(_find_tensors(output)))    else:        self.reducer.prepare_for_backward([])