Padding labels is wrong when using `pad_to_multiple_of`
I haven't tried it myself, but it looks like this line is wrong:

https://github.com/huggingface/transformers/blob/42f359d015aee3835490bdcfa20df657a4d97049/src/transformers/data/data_collator.py#L285

If `self.pad_to_multiple_of` is set to anything but 1, then the length of the labels and the length of the input won't match anymore.
cc @sgugger @Rocketknight1 <|||||>The length of the label and the length of the input rarely match for seq2seq problem, so this is not an issue.<|||||>🤦‍♂️You're right. I was looking for MLM and grabbed the wrong class.