@ptrblck Thank you for the precise answer! 
One last question, is there a way to get the context of the current device (e.g., as in current thread context)? For my intents I can use DDP, but I still have to move data across devices due to the approach I’ve taken of uploading early on GPU. I can see this done by moving from one device to another, I just don’t know if pytorch will currently support it, using tensor move semantics across devices.

Blockquote
The usual work flow would be to try to load the next batch into the RAM while the GPUs are busy training the model to hide the latency of the data loading step.
In each iteration you would push a batch to the device and start loading the next batch.

Is there an example of how to do this? E.g., how to know beforehand the tensors of the batch so I can start uploading before the batch runs through the ops?