The following method is called by DataParallel to create replicas. So the attributes in __dict__ should be replicated as well. But as the flow is “DataParallel forward” -> “replicate models” -> “app model forward”, you need to make sure that the mode is set properly before calling DataParallel forward.

  
      github.com
  
  
    pytorch/pytorch/blob/3001facd7a3942efc7c7e8a42b720b9c884387b3/torch/nn/modules/module.py#L1210-L1219

def _replicate_for_data_parallel(self):
    replica = self.__new__(type(self))
    replica.__dict__ = self.__dict__.copy()

    # replicas do not have parameters themselves, the replicas reference the original
    # module.
    replica._parameters = OrderedDict()
    replica._buffers = replica._buffers.copy()
    replica._modules = replica._modules.copy()
    replica._is_replica = True



  
  
    
    
  
  



I tried myself but I don’t know how to access each replica in multiple GPUs. How to access them?

The replicas are created in every forward pass of DataParallel module. To access them and check, you can modify your model’s forward function and print out the mode value.

  
      github.com
  
  
    pytorch/pytorch/blob/3001facd7a3942efc7c7e8a42b720b9c884387b3/torch/nn/parallel/data_parallel.py#L154-L156 1

replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
outputs = self.parallel_apply(replicas, inputs, kwargs)
return self.gather(outputs, self.output_device)