@sshleifer could you also try using bos as decoder_start_token_id  and modifying adjust_logits_during_generation to return logits as is instead of forcing bos ? If you also get bump in ROUGE score we can confirm the issue. Thanks !