cc @sshleifer
This post might also help

  
    
    
    Facebook/bart-large-cnn has a low rouge score on cnn_dailymail Beginners
  
  
    I had similar issues reproducing the ROUGE scores of facebook/bart-large-cnn on the CNN/Daily Mail dataset. 
First I tried the evaluation script  examples/seq2seq/eval.py with the follwoing arguments: 
export DATA_DIR=cnn_dm
python3 ./run_eval.py
   facebook/bart-large-cnn
   $DATA_DIR/val.source
   dbart_val_generations.txt
   --reference_path $DATA_DIR/val.target
   --score_path cnn_rouge.json
   --task summarization
   --device cuda
   --max_source_length 1024
   --max_target_length 56
   --f…