Well, the only usage of device I am aware of is, when some tensor on the device (e.g. mask) is created on the fly. e.g. buckets for relative positional embeddings
rp_bucket.to(self.relative_attention_bias.weight.device)
The file https://github.com/huggingface/transformers/blob/fedabcd1545839798004b2b468f191ec2244442f/src/transformers/modeling_t5.py contains all the model code.
As for torch nightly, I have installed torch-1.8.0.dev20201029-cp36-cp36m-linux_x86_64, and basically the same thing happened (only without showing c++ code ‘trace’)
2020-10-30 09:06:08,462 INFO root: Epoch 0
  0%|                                                                                         | 0/78424 [00:00<?, ?it/s]0
0
0
1
1
1
2
2
2
3
3
3
2020-10-30 09:06:18,731 INFO root: Finished after 0.1711417039235433 minutes.
2020-10-30 09:06:18,731 ERROR root: Expected all tensors to be on the same device, but found at least two devices, cuda:3 and cuda:0!
2020-10-30 09:06:18,734 ERROR root: Traceback (most recent call last):
  File "fit_qa/scripts/reader/run_generative_train_fuse_dec_multigpu.py", line 97, in <module>
    r = framework.fit()
  File "/home/maskeduser/workaround/QA/fit_qa/scripts/reader/reader/generative_reader_trainer_fusedec.py", line 164, in fit
    scheduler=scheduler)
  File "/home/maskeduser/workaround/QA/fit_qa/scripts/reader/reader/generative_reader_trainer_fusedec.py", line 264, in train_epoch
    raise e
  File "/home/maskeduser/workaround/QA/fit_qa/scripts/reader/reader/generative_reader_trainer_fusedec.py", line 234, in train_epoch
    loss.backward()
  File "/home/maskeduser/miniconda3/envs/qaenv/lib/python3.6/site-packages/torch/tensor.py", line 227, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/maskeduser/miniconda3/envs/qaenv/lib/python3.6/site-packages/torch/autograd/__init__.py", line 138, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:3 and cuda:0!

Traceback (most recent call last):
  File "fit_qa/scripts/reader/run_generative_train_fuse_dec_multigpu.py", line 101, in <module>
    raise be
  File "fit_qa/scripts/reader/run_generative_train_fuse_dec_multigpu.py", line 97, in <module>
    r = framework.fit()
  File "/home/maskeduser/workaround/QA/fit_qa/scripts/reader/reader/generative_reader_trainer_fusedec.py", line 164, in fit
    scheduler=scheduler)
  File "/home/maskeduser/workaround/QA/fit_qa/scripts/reader/reader/generative_reader_trainer_fusedec.py", line 264, in train_epoch
    raise e
  File "/home/maskeduser/workaround/QA/fit_qa/scripts/reader/reader/generative_reader_trainer_fusedec.py", line 234, in train_epoch
    loss.backward()
  File "/home/maskeduser/miniconda3/envs/qaenv/lib/python3.6/site-packages/torch/tensor.py", line 227, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/maskeduser/miniconda3/envs/qaenv/lib/python3.6/site-packages/torch/autograd/__init__.py", line 138, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:3 and cuda:0!
  0%|                                                                                         | 0/78424 [00:12<?, ?it/s]