pegasus-xsum summarized a story of Eiffel Tower into one on the World Trade Center
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.2
- Platform: 
- Python version: 3.7.10
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.4.1
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?:

## Who can help
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj

## Information

The wired thing happened when I tried model pegasus-xsum for text summarization using the example code and data. I noticed that the output describes a similar but obviously different story than the one in the input. I expected to see some description of the Eiffel Tower, but the output is all about New York's World Trade Center!!

I noticed that the online demo version works fine, and the summary output is still on the Eiffel Tower.
https://huggingface.co/google/pegasus-xsum

It appears that pegasus-xsum model in my code generated the summary from some training data, but not the input I gave (retained memory?). How can I git the model behave normally like the online version?

The code I used (adopted from the online demo page):
```
text1="The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct." 

from transformers import AutoTokenizer, TFAutoModelForSeq2SeqLM
tokenizer1 = AutoTokenizer.from_pretrained("google/pegasus-xsum")  
model1 = TFAutoModelForSeq2SeqLM.from_pretrained("google/pegasus-xsum")
inputs1_1 = tokenizer1.encode("summarize: " + text1, return_tensors="tf", max_length=1024)
outputs1_1 = model1.generate(inputs1_1, max_length=150, min_length=40, length_penalty=2.0, num_beams=4, early_stopping=True)
tokenizer1.decode(outputs1_1[0])  
```

the output I got:
`"<pad> New York's World Trade Center is the tallest building in the United States and one of the world's tallest structures, with a total height of 1,776ft (541m), according to Guinness World Records."`

Hey @xiaoda99,

You should **not** append a `summarize: ` prefix for Pegasus.

Running this code:

```python
text1="The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct." 

from transformers import AutoTokenizer, TFAutoModelForSeq2SeqLM
tokenizer1 = AutoTokenizer.from_pretrained("google/pegasus-xsum")  
model1 = TFAutoModelForSeq2SeqLM.from_pretrained("google/pegasus-xsum")
inputs1_1 = tokenizer1.encode(text1, return_tensors="tf", max_length=1024)
outputs1_1 = model1.generate(inputs1_1, max_length=150, min_length=40, length_penalty=2.0, num_beams=4, early_stopping=True)
tokenizer1.decode(outputs1_1[0])  
```

gives me better results: 

```
"<pad> The Eiffel Tower is a free-standing structure in Paris, France, built in 1889 by Gustave Eiffel as a monument to his country's national symbol, the Eiffel Tower, which was later renamed the Louvre."
```<|||||>@patrickvonplaten ,
thanks for the response. The summary you posted is about Eiffel Tower, but the information is not really from the input text. The same problem still exists, it spit out some different story than the one in the input, which is likely from the original training data.

Can you check on why this happens? thanks.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.