Sorry, I’ve tried to make all of inputs to be on the same device.
by changing
return torch.cat(outputs, 1).cuda(dev)

to
return torch.cat([o.cuda(dev) for o in outputs], 1).cuda(dev)

All of tensors will be on cuda:3 before cat.
But it raise a new error. Is it necessary to compute loss separately in each output? (Because it’s in separate device)
RuntimeError: expected device cuda:3 but got device cuda:2

and below is the new anomaly detect warning:
Warning: Traceback of forward call that caused the error:
  File "train.py", line 77, in <module>
    main()
  File "train.py", line 70, in main
    trainer.train()
  File "../../utils/trainer.py", line 186, in train
    self.train_iteration()
  File "../../utils/trainer.py", line 162, in train_iteration
    output = self.model(train_data)
  File "/home/xxx/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "../../models/mrunet/model.py", line 58, in forward
    x = self.outConv(x).cuda(dev)
  File "/home/xxx/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xxx/.local/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 345, in forward
    return self.conv2d_forward(input, self.weight)
  File "/home/xxx/.local/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward
    self.padding, self.dilation, self.groups)
 (print_stack at /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:57)