@sshleifer, could it be due to the adjust_logits  issue ? Just a guess but as I posted there, after modifying the  adjust_logits_during_generation  BLUE-4 score for my model went from 13.09 to 19.14 for bart-base