We can send tensors to different devices. to(device=“cpu:x”)
But how do we get the “cpu:x” device?
The setting is, using MPI to spawn multiple threads for computations (not backprop) and then moving tensors from each thread to a central GPU (for backprop) and back to each CPU.