Since you call Tensor.to in the body of the forward function it will always live on that device. The DataParallel module does the following for you: replicate your model across N devices, split your input tensor(s) in N chunks (across dimension 0 by default), run forward on each of the replicas, and concatenate the result. Every model replica where the forward function is called is already pinned to a device, so you shouldn’t explicitly call to(device) on intermediate tensors.