My question is that why don’t we need to transfer the inputs to gpus ?

Thanks for pointing this out. Yep, it’s a good practice to also move the inputs to the destination GPU. The reason that the tutorial code didn’t fail is because DDP will recursively detect tensors in the inputs and move them to the target device. See the code below:

  
      github.com
  
  
    pytorch/pytorch/blob/7df84452423f44ebe1db40a2e3463066bf954f95/torch/nn/parallel/distributed.py#L683-L684 1

inputs, kwargs = self.to_kwargs(inputs, kwargs, self.device_ids[0])output = self.module(*inputs[0], **kwargs[0])