The tensorflow implementation of T5ForConditionalGeneration runs much slower than the pytorch one. GPU utilization is 30%
# 🐛 Bug
When I am running the official example in examples/summarization/t5/example on PyTorch, I have much better performance than the Tensorflow one. When running on PyTorch it needs 4s per iteration and uses 100% of the GPU. When running the TensorFlow model it needs 30s per iteration and the GPU utilization is  15-20%.
## Information

Model I am using: t5-small

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (give the name): CNN Dailymail
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
1. Follow the steps in the official examples/summarization/t5/example
2. Use the modified evaluate_cnn.py script provided below

```
import argparse
from pathlib import Path

from tqdm import tqdm

from rouge_score import rouge_scorer, scoring
from transformers import TFT5ForConditionalGeneration, T5Tokenizer


def chunks(lst, n):
    """Yield successive n-sized chunks from lst."""
    for i in range(0, len(lst), n):
        yield lst[i : i + n]

def generate_summaries(lns, output_file_path, model_size, batch_size):
    output_file = Path(output_file_path).open("w")

    model = TFT5ForConditionalGeneration.from_pretrained(model_size)

    tokenizer = T5Tokenizer.from_pretrained(model_size)

    # update config with summarization specific params
    task_specific_params = model.config.task_specific_params
    if task_specific_params is not None:
        model.config.update(task_specific_params.get("summarization", {}))

    for batch in tqdm(list(chunks(lns, batch_size))):
        batch = [model.config.prefix + text for text in batch]
        dct = tokenizer.batch_encode_plus(batch, max_length=512, return_tensors="tf", pad_to_max_length=True)
        input_ids = dct["input_ids"]#.to(device)
        attention_mask = dct["attention_mask"]#.to(device)

        summaries = model.generate(input_ids=input_ids, attention_mask=attention_mask) 
        dec = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summaries]
        for hypothesis in dec:
            output_file.write(hypothesis + "\n")
            output_file.flush()


def calculate_rouge(output_lns, reference_lns, score_path):
    score_file = Path(score_path).open("w")
    scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True)
    aggregator = scoring.BootstrapAggregator()

    for reference_ln, output_ln in zip(reference_lns, output_lns):
        scores = scorer.score(reference_ln, output_ln)
        aggregator.add_scores(scores)

    result = aggregator.aggregate()
    score_file.write(
        "ROUGE_1: \n{} \n\n ROUGE_2: \n{} \n\n ROUGE_L: \n{} \n\n".format(
            result["rouge1"], result["rouge2"], result["rougeL"]
        )
    )


def run_generate():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "model_size",
        type=str,
        help="T5 model size, either 't5-small', 't5-base', 't5-large', 't5-3b', 't5-11b'. Defaults to 't5-base'.",
        default="t5-base",
    )
    parser.add_argument(
        "input_path", type=str, help="like cnn_dm/test_articles_input.txt",
    )
    parser.add_argument(
        "output_path", type=str, help="where to save summaries",
    )
    parser.add_argument("reference_path", type=str, help="like cnn_dm/test_reference_summaries.txt")
    parser.add_argument(
        "score_path", type=str, help="where to save the rouge score",
    )
    parser.add_argument(
        "--batch_size", type=int, default=8, required=False, help="batch size: how many to summarize at a time",
    )
    parser.add_argument(
        "--no_cuda", default=False, type=bool, help="Whether to force the execution on CPU.",
    )

    args = parser.parse_args()
    # args.device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")

    source_lns = [x.rstrip() for x in open(args.input_path).readlines()]

    generate_summaries(source_lns, args.output_path, args.model_size, args.batch_size)

    output_lns = [x.rstrip() for x in open(args.output_path).readlines()]
    reference_lns = [x.rstrip() for x in open(args.reference_path).readlines()]

    calculate_rouge(output_lns, reference_lns, args.score_path)


if __name__ == "__main__":
    run_generate()
```


## Expected behavior

The Tensorflow code   should work with similar  performance as the PyTorch one

## Environment info
absl-py                  0.9.0              
astor                    0.7.1              
attrs                    19.3.0             
blinker                  1.4                
boto3                    1.12.34            
botocore                 1.15.34            
cachetools               3.1.1              
certifi                  2019.11.28         
cffi                     1.14.0             
chardet                  3.0.4              
click                    7.1.1              
cryptography             2.8                
dill                     0.3.1.1            
docutils                 0.15.2             
filelock                 3.0.12             
future                   0.18.2             
gast                     0.2.2              
google-auth              1.12.0             
google-auth-oauthlib     0.4.1              
google-pasta             0.2.0              
googleapis-common-protos 1.51.0             
grpcio                   1.27.2             
h5py                     2.10.0             
idna                     2.9                
jmespath                 0.9.5              
joblib                   0.14.1             
Keras-Applications       1.0.8              
Keras-Preprocessing      1.1.0              
Markdown                 3.2.1              
nltk                     3.4.5              
numpy                    1.18.1             
oauthlib                 3.0.1              
opt-einsum               3.2.0              
pip                      20.0.2             
promise                  2.3                
protobuf                 3.11.4             
pyasn1                   0.4.8              
pyasn1-modules           0.2.7              
pycparser                2.20               
PyJWT                    1.7.1              
pyOpenSSL                19.1.0             
PySocks                  1.7.1              
python-dateutil          2.8.1              
regex                    2020.2.20          
requests                 2.23.0             
requests-oauthlib        1.2.0              
rouge-score              0.0.3              
rsa                      4.0                
s3transfer               0.3.3              
sacremoses               0.0.38             
scipy                    1.4.1              
sentencepiece            0.1.85             
setuptools               46.1.3.post20200325
six                      1.14.0             
tensorboard              2.1.0              
tensorflow               2.1.0              
tensorflow-datasets      2.1.0              
tensorflow-estimator     2.1.0              
tensorflow-gpu           2.1.0              
tensorflow-metadata      0.21.1             
termcolor                1.1.0              
tokenizers               0.5.2              
torch                    1.4.0              
tqdm                     4.45.0             
transformers             2.7.0              
urllib3                  1.25.7             
Werkzeug                 1.0.1              
wheel                    0.34.2             
wrapt                    1.12.1 
     
- `transformers` version: 2.70, 2.7.1, and builded from 81484b447b7d8504ff5e1cfff38ec35918383963
- Platform: Ubuntu Ubuntu 18.04.4 LTS
- Python version: 3.7.6
- PyTorch version (GPU?): 
- Tensorflow version (GPU?):2.1.0
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?:No

I want to take a closer look in a week or so at this. This issue seems to be related: https://github.com/huggingface/transformers/issues/4634<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>The problem is likely because of `generate()` not being compatible with `tf.function`. I want to take a look at this in more detail while working on this PR: https://github.com/huggingface/transformers/pull/5662<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
