Not really my findings, just the pegasus paper and a bug fix. We used to be forcing Bart and all it’s children (Mbart, Pegasus) to generate bos after decoder_start_token_id, because fairseq does that.
With @valhalla 's help, we figured out that this only helps for bart-large-cnn, so I added a config attribute force_bos_token_to_be_generated, which defaults to False, but is set to true for bart-large-cnn and distilbart-cnn*.
Metrics Impact
facebook/bart-large-xsum : + 0.6 ROUGE2,  22.38
facebook/mbart-large-enro:  +0.3 BLEU,  28.15
google/pegasus-xsum:  +.3 ROUGE2, 24.43
facebook/bart-large-cnn: 0