Hi!
I am having the same problem trying to run a module on 2 GPUs. I tried changing every cuda() reference to cuda(device_id=0) but I’m still getting the same error.
RuntimeError: tensors are on different GPUs
I am not sure I even understand the reason why this error happens, but would you mind telling me why moving all cuda tensors to GPU 0 fixes it in your case?