Hello, thank you for you help.

make sure that all the inputs are properly on the same GPU.

The inputs and the labels(for calculating losses) are in device[cuda:3].

I’ve enabled the anomaly detection by adding this line.
torch.autograd.set_detect_anomaly(True)

The warning was appeared as below:
Warning: Traceback of forward call that caused the error:
  File "train.py", line 77, in <module>
    main()
  File "train.py", line 70, in main
    trainer.train()
  File "../../utils/trainer.py", line 186, in train
    self.train_iteration()
  File "../../utils/trainer.py", line 162, in train_iteration
    output = self.model(train_data)
  File "/home/xxx/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "../../models/model.py", line 64, in forward
    return torch.cat(outputs, 1).cuda(dev)
 (print_stack at /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:57)

Thank you.