@sshleifer
Isn’t it necessary to add a start token to the decoder sequence (be it anything eos, bos, pad or whatever else)  when training generative models, because the labels need to be shifted  to calculate loss, without shifting the labels the models will only learn to produce the last token it has seen.
Is this right ?
And I think MBart and PEGASUS don’t have bos  and may have been pre-trained by using pad token (like T5) as the first token which could explain this behavior. I couldn’t find this info in PEGASUS repo. What was the decoder_start_token_id when pre-training PEGASUS and MBart ?