From print(trainer.args.device) it prints xla:0 so it seems it is definitely on the TPU, the trainer anyways. Here is the printout of the dictionary that I return:
Return Dict {'input_ids': tensor([[743,  10,   1,  ...,   0,   0,   0],
        [743,  10,  96,  ...,   0,   0,   0],
        [743,  10,   1,  ...,   0,   0,   0],
        [743,  10,   1,  ...,   0,   0,   0]], device='xla:0'), 'attention_mask': tensor([[1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0]], device='xla:0'), 'lm_labels': tensor([[32104, 32103,  2490,  ...,  -100,  -100,  -100],
        [32104, 32103,  2490,  ...,  2281,   599,    31],
        [32104, 32103,  2490,  ...,  -100,  -100,  -100],
        [32104, 32104,  2490,  ...,    17, 18392,    76]], device='xla:0'), 'decoder_attention_mask': tensor([[1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 1, 1, 1],
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 1, 1, 1]], device='xla:0')}

I think it has to do with me places the model on the CPU cpu_model = self.model.to('cpu') to perform inference. However, I have to do this because there is some other bug that doesn’t let me use the .generate function on TPUs. I’m gonna try just using a regular old GPU and see if that helps at all.