Unable to reproduce gigawords results from google/pegasus-gigaword
@patrickvonplaten

Hi Patrick,

I am trying to reproduce the PEGASUS results for gigaword. I used gigawords in datasets library and directly use its testing portion without further preprocessing. I used PegasusForConditionalGeneration, PegasusTokenizer (I used checkpoint from Google, google/pegasus-gigaword) to decode summary using default setting. However, my ROUGE score is bit deviated from the original paper reported (my results roug1,2,L: 28/12/25 vs 39.65/20.47/36.76). I wondered if my setup was incorrect. 
Hey @xu1998hz,

We're trying to keep Transformers issues for bugs in the core library. Could you try to use the forum instead: https://discuss.huggingface.co/ ? :-)<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.