I have another question that does data copy between GPUs in DataParallel module utilize the same kernel?

Yes. DataParallel calls into scatter. Its C++ implementation is linked below. It’s basically calling Tensor.to in a loop

  
      github.com
  
  
    pytorch/pytorch/blob/89c0efb30b5bad49e717c1b1a797bac3c62e8b7e/torch/csrc/cuda/comm.cpp#L201-L206 4

chunks[chunk] =
    chunks[chunk].to(
        {DeviceType::CUDA, device_index},
        /*non_blocking=*/true,
        /*copy=*/false,
        /*memory_format=*/at::MemoryFormat::Preserve);