Unable to reduce time in summarization!
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:  `t5-large`
- Platform:  Amazon Sagemaker
- Python version: 3.7
- Tensorflow version (GPU?):2.3
- Using distributed or parallel set-up in script?:  Unable to implement it properly.

### Who can help
@LysandreJik, @patil-suraj, @jplu

### Problem:
Unable to reduce the summarization time.

The tasks I am working on is:

I am using pretrained transformer of T5(`TFT5ForConditionalGeneration`)  for text summarization.
Brief script:
```
inputs = tokenizer("summarize: " + text, return_tensors="tf").input_ids

outputs = model.generate(
        inputs, 
        max_length=200, 
        min_length=5,  
        num_beams=5,)
```
I tried to use distributed strategy of tensorflow. But it doesn't made any improvement.
```
strategy = tf.distribute.MirroredStrategy()
  
strategy = tf.distribute.MirroredStrategy(devices=["/gpu:0", "/gpu:1"])

```

## Expected behavior
I am hoping that if we increase the number of **GPU**, time must be reduced. But It is not happening in this case.

Hello!

The generation part of TensorFlow can only be run on eagermode, hence doesn't matter how you execute it, you will not be able to run it "fast". It is planned to bring a graph execution for the generation, but no ETA yet. Sorry for the inconvenience.