The labels are popped because otherwise your model computes the losses twice, so two SoftMaxes, which is a very heavy operation. You need to pass along the decoder_input_ids when you want to use label smoothing with the Trainer, as generated by DataCollatorForSeq2Seq.