strange memory usage for t5 models
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.3
- Platform: linux
- Python version: 3.7
- PyTorch version (GPU?): yes 
- Tensorflow version (GPU?): - 
- Using GPU in script?: no 
- Using distributed or parallel set-up in script?: no 

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
 t5: @patrickvonplaten, @patil-suraj

## Information

Hi
I am having a hard time with training  t5 models for classification using seq2seq examples on paws-x dataset, I am often getting out of memory error for even small batch sizes, and there must be a bug in seq2seq model with t5 causing large  usage of memory, thanks for having a look

```
Traceback (most recent call last):
  File "run_seq2seq.py", line 593, in <module>
    main()
  File "run_seq2seq.py", line 551, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/users/dorood/seq2seq/third_party/trainers/trainer.py", line 321, in train
    tr_loss += self.training_step(model, inputs)
  File "/users/dorood/libs/anaconda3/envs/test2/lib/python3.7/site-packages/transformers/trainer.py", line 1485, in training_step
    loss = self.compute_loss(model, inputs)
  File "/users/dorood/libs/anaconda3/envs/test2/lib/python3.7/site-packages/transformers/trainer.py", line 1517, in compute_loss
    outputs = model(**inputs)
  File "/users/dorood/libs/anaconda3/envs/test2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/users/dorood/seq2seq/third_party/models/t5/modeling_t5.py", line 1751, in forward
    lang=lang
  File "/users/dorood/test2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/users/dorood/seq2seq/third_party/models/t5/modeling_t5.py", line 1115, in forward
    task=task
  File "/users/dorood/test2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/users/dorood/seq2seq/third_party/models/t5/modeling_t5.py", line 752, in forward
    output_attentions=output_attentions,
  File "/users/dorood/libs/anaconda3/envs/test2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/users/dorood/seq2seq/third_party/models/t5/modeling_t5.py", line 653, in forward
    output_attentions=output_attentions,
  File "/users/dorood/libs/anaconda3/envs/test2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/users/dorood/seq2seq/third_party/models/t5/modeling_t5.py", line 557, in forward
    attn_output = unshape(torch.matmul(attn_weights, value_states))  # (batch_size, seq_length, dim)
RuntimeError: CUDA out of memory. Tried to allocate 42.00 MiB (GPU 0; 23.70 GiB total capacity; 21.14 GiB already allocated; 1.69 MiB free; 22.36 GiB reserved in total by PyTorch)

```
Which model are you using? What's the command you're using to launch the training?<|||||>Hi
I will close this issue and open up a proper reporting as I see the issue is arising from loading a checkpointin trainer class 