Hey @ayalaa2, DistributedDataParallel’s (DDP) ctor would go through all parameters and skip the ones whose requires_grad=False. So, there won’t be communication on those grad, but you will have to set their require_grad field before passing it to DDP. After the ctor, changing the requires_grad attribute makes no difference. See the code below in DDP ctor.

  
      github.com
  
  
    pytorch/pytorch/blob/6bd88f581a6323d026e08b65ffee75bfe162501f/torch/nn/parallel/distributed.py#L456-L464 14

# Build tuple of (module, parameter) for all parameters that require grads.
modules_and_parameters = [
    [
        (module, parameter)
        for module in replica.modules()
        for parameter in filter(
            lambda parameter: parameter.requires_grad,
            parameters(module, recurse=False))
    ] for replica in self._module_copies]



  
  
    
    
  
  


Another way to skip communication is to use the no_sync 7 context manager.