BART.generate: possible to reduce time/memory?
# 🐛 Performance issues

I did a quick benchmark between HuggingFace's implementation of **BART** and FairSeq's implementation.

You can find the benchmark code [here](https://gist.github.com/Colanim/cc418d19e5e107f462bac306f53ba994).

---

Here is my results, on a single GPU GTX 1080 (12 GiB of memory) :

| FP16 - Batch size 16 | s/batch | s/sample |
|----------------------|---------|----------|
| FairSeq              | 8.8676  | 0.5664   |
| HuggingFace          | 12.3358 | 0.7879   |

| FP16 - Batch size 32 | s/batch | s/sample |
|----------------------|---------|----------|
| FairSeq              | 17.1247 | 0.5469   |
| HuggingFace          | OOM     | OOM      |

| FP16 - Batch size 1 | s/sample |
|---------------------|----------|
| FairSeq             | 1.6743   |
| HuggingFace         | 1.8856   |

| FP32 - Batch size 1 | s/sample |
|---------------------|----------|
| FairSeq             | 1.7865   |
| HuggingFace         | 2.0670   |

---

**FairSeq is consistently faster than HuggingFace on all my experiments.**

---

This sparks a few questions :

* Do you have similar results on your side ? Did I mess my benchmark ?
* Why HuggingFace's implementation is significantly slower ?
* Why HuggingFace's implementation takes more space in memory (illustrated by `OOM` with batch size of 32) ?
* Is the release of the `Summarization Pipeline` going to improve this ?

@sshleifer 

 
1) Identical to my benchmark for speed. Hadn't tested memory but I'm not surprised that their implementation is less.

For both memory and speed, they have a lot of clever tricks that we haven't implemented yet.

4) Summarization Pipeline will not help, but I will take a longer look at this tomorrow and see if we can improve.
<|||||>On master, the gap has closed considerably!
<16GB GPU RAM for fp16, bs=32, and timings much closer:
![image](https://user-images.githubusercontent.com/6045025/77931576-55d7bd00-727a-11ea-8904-e57070635087.png)

My numbers are a bit lower than yours because I am on an NVIDIA RTX GPU.
<|||||>I tested again and I have similar results ! Thanks for the fix.

I now have exact same GPU memory utilization.

---

About the (now) small difference of inference time between implementations, do you know from where it comes from ?<|||||>Haven't investigated. So far, I just investigated memory and the speed improvements were a happy side effect.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
