Hi,


You can use tensor.device. It returns you the current device of the tensor, and the index of the GPU if the device is cuda.


If model and input are not in the same device, it will crash. You cannot do operations between tensors in cpu and tensors in cuda.


Exemple:
In [12]: torch.rand(10, 10).cuda() + torch.rand(10, 10)                                                                                                                                                            
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-12-4280bdc898ed> in <module>
----> 1 torch.rand(10, 10).cuda() + torch.rand(10, 10)
RuntimeError: expected device cuda:0 and dtype Float but got device cpu and dtype Float


In DataParallel, the batch will be split among the gpus. At the end, the gradients will be sent back to the host and summed, and the step happens here.
So every device is hosting the whole model, plus a portion of the batch.