Can you post the full error trace? Just to be safe, you can do
        return {
            'input_ids': input_ids.to(xm.xla_device()), 
            'attention_mask': attention_mask.to(xm.xla_device()),
            'lm_labels': lm_labels.to(xm.xla_device()), 
            'decoder_attention_mask': decoder_attention_mask.to(xm.xla_device())
        }