Thanks for your quick reply ptrblck!
It took me some time but I found the error, one of my tensors was in CPU (decoder_start_token_embed). I was inserting that tensor into another one which was in GPU:
shifted_inputs_embeds[:, 0, :] = decoder_start_token_embed

The error message on the backward pass didn’t shed much light on what that tensor would be. It would be great to have a better way to debug such errors.