Hi, I’m using allennlp to do distributed bert training.
In their code, model has some customized functions, e.g., get_metrics, and  get_regularization_penalty. After wrapping it with ddp, there is a comment says
        # Using `DistributedDataParallel`(ddp) brings in a quirk wrt AllenNLP's `Model` interface and its
        # usage. A `Model` object is wrapped by `ddp`, but assigning the wrapped model to `self.model`
        # will break the usages such as `Model.get_regularization_penalty`, `Model.get_metrics`, etc.
        #
        # Hence a reference to Pytorch's object is maintained in the case of distributed training and in the
        # normal case, reference to `Model` is retained. This reference is only used in
        # these places: `model.__call__`, `model.train` and `model.eval`.


  

      github.com
  

  
    allenai/allennlp/blob/c5bff8ba0d835eb03931f10f4f427ffe936cf796/allennlp/training/gradient_descent_trainer.py#L302 2



      
      
          self._num_gradient_accumulation_steps = num_gradient_accumulation_steps
          
          
# Enable automatic mixed precision training.
          self._scaler: Optional[amp.GradScaler] = None
          self._use_amp = use_amp
          if self._use_amp:
              if self.cuda_device == torch.device("cpu"):
                  raise ValueError("Using AMP requires a cuda device")
              self._scaler = amp.GradScaler()
          
          
# Using `DistributedDataParallel`(ddp) brings in a quirk wrt AllenNLP's `Model` interface and its
          # usage. A `Model` object is wrapped by `ddp`, but assigning the wrapped model to `self.model`
          # will break the usages such as `Model.get_regularization_penalty`, `Model.get_metrics`, etc.
          #
          # Hence a reference to Pytorch's object is maintained in the case of distributed training and in the
          # normal case, reference to `Model` is retained. This reference is only used in
          # these places: `model.__call__`, `model.train` and `model.eval`.
          if self._distributed:
              self._pytorch_model = DistributedDataParallel(
                  self.model,
                  device_ids=None if self.cuda_device == torch.device("cpu") else [self.cuda_device],
      
    
  


  

  
    
    
  

  


My question is what is the relationship between self.model and its wrapped version self._pytorch_model?
Do they share parameters and runtime state?