Hello,
I’m trying to load data in separate GPUs, and then run multi-GPU batch training.
I’ve managed to balance data loaded across 8 GPUs, but once I start training, I trigger an assertion:
RuntimeError: Assertion `THCTensor_(checkGPU)(state, 5, input, target, weights, output, total_weight)' failed. Some of weight/gradient/input tensors are located on different GPUs. Please move them to a single one. at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:24

This is understandable: the data is already spread across various devices.
Is this not supported? I understand that multi-GPU batch will do that during the iteration of the batch,
so is the only way to achieve this, to copy across different devices, on the device where the batch is being used?
How can I get the device where the batch will be placed? I know that I can query the device on which the current input/sample tensor is, and do a copy from one to another (although I’m not sure this is ideal) provided that forces the batch and the tensors to reside in the same device.
Also, would that create a bottleneck?
EDIT:
I just realized that there exists a method called nn.parallel.scatter and other similar ones, which I am going to assume achieve the same functionality?