T5 doubling training time per iteration from save_steps to save_steps (1st 100 steps 33s/it - then, 75s/it)
## Environment info

- `transformers` version: 4.3.0.dev0
- Platform: Ubuntu 18.04
- Python version: 3.7
- PyTorch version (GPU?): 1.7.1 (YES)
- Tensorflow version (GPU?):
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?:

### Who can help
@patrickvonplaten @patil-suraj @sgugger

## Information

Model I am using (Bert, XLNet ...): T5

The problem arises when using:
* [x] the official example scripts: (give details below)
The problem arises when using examples/seq2seq/run_seq2seq.py
* [ ] my own modified scripts: (give details below)

The problem arises when trying to fine-tune t5 in a text2text task using the official scripts (run_seq2seq.py). The first problem is that after the first checkpoint saved, the training becomes super slow, taking twice the time it took until that checkpoint. The second problem is that when you try to load a model from one of those checkpoints, it's like the model has increased in size (not directly, but when processing batches it uses much more memory). Let me explain myself. If you start with t5-large, in a P100 16GB gpu I can fit around 350 sequence length, 52 target length, 2 train batch size per device. However, if I start from one of the checkpoints saved (which are also t5-large, just a little bit more trained) I cannot even fit batch size 1, seq length 256, target length 50, and this is really strange.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
A summarization dataset

## To reproduce

Steps to reproduce the behavior:

1. Get any summarization or any text2text dataset.
2. Train a t5-large using examples/seq2seq/run_seq2seq.py and set save_steps to 100
3. Wait until the 100 first steps have been completed, you'll see that with a batch size of 64 (using gradient accumulation) it takes 33s/iter approx.
4. After that, when the first checkpoint is created, the training will re-start, but this time using 75s/iter, doubling its time. Over the course of the rest of the training, you'll see this same timing, it never goes back to 33s/iter. 
5. Then, try to adapt this model to another dataset or to the same dataset itself, and appreciate that it's unfeasible to train with the previously specified parameters, which shouldn't happen because this model should use the same memory as the original t5. However, it's not that the model itself is bigger when loaded in the gpu, what happens is that when loading batches of tensors and processing them, memory requirements exceed significantly the ones required by the same training setup changing my trained model for t5-large (the only thing that changes is the model).

## Expected behavior

It's expected that training time (iterations /s) is approx. the same during the whole training process, it's not supposed to double due to (apparently) no reason. Moreover, it's expected that when you load a re-trained t5-large you can fine-tune it with the same training setup (batch size etc) as the one used for t5-large.
Hey @alexvaca0, 

Could you please provide us the command you used to train `t5-large` with `run_seq2seq.py` so that we can reproduce? <|||||>Also note that the timing is just a tqdm timing, that takes into account how long each iteration takes. Since the model saving takes a long time, it's normal to see it jump around a saving step and never go back to the best score of the first iterations.<|||||>I don't intend to see it go back to the best score of the first iterations, but long after the last time it has been saved, it still takes twice as much time per iteration. Let me share with you the command used:

First, I pre-trained t5 using the concepts explained here (taken from https://huggingface.co/transformers/master/model_doc/t5.html):

```{python}
input_ids = tokenizer('The <extra_id_0> walks in <extra_id_1> park', return_tensors='pt').input_ids
labels = tokenizer('<extra_id_0> cute dog <extra_id_1> the <extra_id_2>', return_tensors='pt').input_ids
# the forward function automatically creates the correct decoder_input_ids
loss = model(input_ids=input_ids, labels=labels).loss
```

For that I used the following command:

```{bash}
nohup python transformers/examples/seq2seq/run_seq2seq.py \
      --model_name_or_path  t5-large \
      --do_eval --do_train \
      --train_file /perturbed_data/tr.csv \
      --validation_file  /perturbed_data/val.csv \
      --output_dir t5_lm \
      --overwrite_output_dir \
      --per_device_train_batch_size=2 \
      --per_device_eval_batch_size=16 \
      --eval_accumulation_steps=10 \
      --max_source_length 346 \
      --max_target_length 60 \
      --val_max_target_length 60 --evaluation_strategy steps \
      --gradient_accumulation_steps 128 --num_train_epochs=20 --eval_beams=1 \
      --load_best_model_at_end --save_steps 100 --logging_steps 100  --learning_rate 7e-5 > bart_basic.txt &
```

Then, with that model trained, I tried to fine-tune it on a summarization task with the following command:

```{bash}
python transformers/examples/seq2seq/run_seq2seq.py \
      --model_name_or_path  t5-lm \
      --do_eval --do_train \
      --train_file summary_train_df.csv  \
      --validation_file summary_val_df.csv \
      --output_dir t5_0802  \
      --overwrite_output_dir \
      --per_device_train_batch_size=2 \
      --per_device_eval_batch_size=16 \
      --eval_accumulation_steps=10 \
      --max_source_length 346 \
      --max_target_length 60 \
      --val_max_target_length 60 --evaluation_strategy steps \
      --gradient_accumulation_steps 32 --num_train_epochs=20 --eval_beams=1 \
      --load_best_model_at_end --save_steps 250 --logging_steps 250  --learning_rate 3e-5
```

This is for the second part of the issue, that is, the fact that when trying to load my lm-trained t5 from disk and re-train it on a summarization corpus the batches it processes suddenly occupy much more than when they're processed by t5-large (without re-training on the LM task). 

For the other part of the issue, the only command that needs to be run is:

```{bash}
python transformers/examples/seq2seq/run_seq2seq.py \
      --model_name_or_path  t5-large \
      --do_eval --do_train \
      --train_file summary_train_df.csv  \
      --validation_file summary_val_df.csv \
      --output_dir t5_0802  \
      --overwrite_output_dir \
      --per_device_train_batch_size=2 \
      --per_device_eval_batch_size=16 \
      --eval_accumulation_steps=10 \
      --max_source_length 346 \
      --max_target_length 60 \
      --val_max_target_length 60 --evaluation_strategy steps \
      --gradient_accumulation_steps 32 --num_train_epochs=20 --eval_beams=1 \
      --load_best_model_at_end --save_steps 250 --logging_steps 250  --learning_rate 3e-5
```

@patrickvonplaten @sgugger <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.