Ah yes, this can’t work because your model returns logits that have seq_length as a dimension. Trainer tries to concatenate all predictions and can’t do that since they have different shapes. So for evaluation, you need to pad to the max length. (This wouldn’t be the case if you used a model that doesn’t return logits having the sequence length in their dimensions, like a classification model).