Hello,
I am in a very similar situation where I have a single node and 8 GPUs.  I used the following resource as a guideline for distributed parallel training: https://github.com/dnddnjs/pytorch-multigpu 20
I was able to run this example fine, but when I try to load the model, I get the following error:
Expected tensor for argument #1 ‘input’ to have the same device as tensor for argument #2 ‘weight’; but device 4 does not equal 0 (while checking arguments for cudnn_convolution)
Could this be a problem in how I am loading the training data?