I met this error as well… I used the padding mask tensor to filter out non-padding result: tensor[padding_mask], after I changed the padding_mask device to cuda as well, the error disappeared, but the final result seems wrong.