PEGASUS generation/decoding VERY Slow
## Environment info
- `transformers` version: 3.4.0
- Platform: Linux-4.19.112+-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.7.0+cu110 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@sshleifer 
 
## Information

Model I am using: *PEGASUS-large*, *PEGASUS-cnn_dailymail*, *PEGASUS-xsum*

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)


The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

I compared the generation/decoding performance of BART and PEGASUS both loaded via AutoModelSeq2SeqForLM fine-tuned on a custom dataset for the same amount of time. Generation was done using basic greedy decoding. PEGASUS models are anywhere between 5-15x slower than BART. Fine-tuning speed was on-par for both models.

## To reproduce

Steps to reproduce the behavior:
1. Load PEGASUS and BART from any of the mentioned checkpoints (using AutoModelSeq2SeqForLM)
1. Fine-tune models
2. Decode using greedy decoding
3. Compare fine-tuning performance with other Seq2Seq Model (BART-large)


## Expected behavior
Decoding performance on-par with BART-large?

this is expected behavior. pegasus has more layers and higher num_beams than BART.
try using num_beams=2 for both models, and or using a distilled model for faster inference. BART also works in fp16.<|||||>I used greedy decoding when generating outputs, so the number of beams should not matter in this case.
Also, PEGASUS is 30% larger than BART, so why does it translate to at least 5x performance drop at inference?<|||||>I can't investigate deeply without a code snippet, but a 5X difference in greedy decoding time for roughly the same number of output characters is larger than I would have expected.

I'd try to verify if the finetuning is, in fact, a part of this problem or whether you're generation parameters/data can demonstrate the slowdown using public models, e.g.

I just ran:

```python
from transformers import *
import time 
torch_device = 'cuda'
for mname in ['facebook/bart-large-xsum', 'google/pegasus-xsum']:
    model = AutoModelForSeq2SeqLM.from_pretrained(mname).to(torch_device)
    tok = AutoTokenizer.from_pretrained(mname)
    batch = tok(['I am a small frog'], return_tensors='pt').to(torch_device)
    t0 = time.time()
    model.generate(**batch, min_length=0, max_length=10, num_beams=1)
    runtime = time.time() - t0
    print(f'{mname}: {runtime:.3f}')
```

and got
```
facebook/bart-large-xsum: 0.089
google/pegasus-xsum: 0.116
```
which is a < 30% difference.<|||||>In our case, the model is part of a larger system, so can't share the code I am using. 
However, I will create a minimal example that allows me to reproduce this issue (most likely by Monday).

One difference I already see is that in our code we work with long sequences that max out the model's limits both on the input and output side. I just started another batch of generation and on the exact same input data BART-large is projected to finish in 45 minutes, PEGASUS-large in 14 hours, both models running on A100 GPUs (one per model).


Edit:
@sshleifer running the script you provided gives the following (avg. across 5 runs):
```
facebook/bart-large-xsum: 0.570
google/pegasus-xsum: 0.952
```
<|||||>Cool.
+ If you want smaller models you can checkout the distilbart/distill-pegasus variants [here](https://huggingface.co/models?search=sshleifer%2Fdistil) [Paper w Details](https://arxiv.org/abs/2010.13002)
+ `bart-large/pegasus-large` probably can't generate very well (unless you've fine-tuned them)<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
