Low ROUGE scores for BertSum
Great work, very easy to pick up and play with. I downloaded the CNN/DM stories from the link provided and selected only the files that belong to the test set following See et al.'s dataset splits (https://github.com/abisee/cnn-dailymail/blob/master/url_lists/all_test.txt). Then I ran the model using the first command provided in the readme.

My question is, what is the expected ROUGE F1 scores on the test set? I expected something near what was presented in the paper for BertSumExtAbs, which is:
R-1: 0.4213
R-2: 0.1960
R-L: 0.3918
But the ROUGE scores I got were much lower:

****** ROUGE SCORES ******

** ROUGE 1
F1        >> 0.303
Precision >> 0.328
Recall    >> 0.288

** ROUGE 2
F1        >> 0.185
Precision >> 0.210
Recall    >> 0.172

** ROUGE L
F1        >> 0.335
Precision >> 0.356
Recall    >> 0.320


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>I'm struggling with this also :( did you make any progress diagnosing the problem?<|||||>No, I did not...I decided to go with a different model<|||||>hello.. guys.. any answers to this? Why is there such a low score? I looked at the summaries, and they seem to be good, but i have no comparison benchmark. However, the rouge scores are much lower than paper. how so? <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>active it<|||||>@AI678 The BertSum is unfortunately not maintained anymore. If you're looking to do summarization, please check out the [seq2seq](https://github.com/huggingface/transformers/tree/master/examples/seq2seq) scripts.