GPU memory consumption increases while training
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2
- Platform: Google Colab
- Python version: 3.6
- PyTorch version (GPU?):  1.6 
- Tensorflow version (GPU?):
- Using GPU in script?: 
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 @LysandreJik @sgugger 

 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger 
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 tensorflow: @jplu 
documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): XLM Multi-lingual

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)
Please see below steps to reproduce

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Importing Python Libraries and preparing the environment
```python
!pip install git+https://github.com/huggingface/transformers
from transformers import ( 
    AutoTokenizer, 
    AutoConfig, 
    AutoModelForSequenceClassification
)
from torch import cuda
device = 'cuda' if cuda.is_available() else 'cpu'
```
2. Loading a pretrained model "xlm-mlm-tlm-xnli15-1024"

```python

MODEL_NAME_OR_PATH = 'xlm-mlm-tlm-xnli15-1024'
CACHE_DIR='cache'
config = AutoConfig.from_pretrained(
    MODEL_NAME_OR_PATH,
    num_labels=7,
    cache_dir=CACHE_DIR,
)
tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME_OR_PATH,
    cache_dir=CACHE_DIR,
)
model = AutoModelForSequenceClassification.from_pretrained(
    MODEL_NAME_OR_PATH,
    from_tf=bool(".ckpt" in MODEL_NAME_OR_PATH),
    config=config,
    cache_dir=CACHE_DIR
)
```
3. Check GPU usage

```
!nvidia-smi
```

4. Moving the model to CUDA

```
model.to(device)
```

then check GPU usage again

```
!nvidia-smi
```

5. Creating test inputs

```python
texts = [
         "aloe vera , wassernabelkrautextrakt , ackerschachtelhalm extrakt , geranium extract , dandelion extract , natriummethyl two sulfolaurate dinatrium two sulfolaurate , sodium cocoyl isethionat , cocamidopropylbetain , cocamidopropylhydroxysultain , kokosglucoside , natrium chlorid , glyceryl oleat , natriumenzoat , guar hydroxypropyltrimonium chloride , tetrasodium glutamat diacetat , decyl glucoside , sodium levulinate , hydroxamsäure , sodium pca , caprylyl glycol , zitronensäure , als koscher zertifizierte pflanzliches glycerin , eukalyptusöl , pfefferminzöl , zitronengrassöl . zertifiziert als organisch . wir verwenden nur die besten natürlichen zutaten . wenn möglich , verwenden wir onezerozero percent zertifizierte organische zutaten und niemals : petrochemikalien , sulfate , parabene , phthalate oder synthetische duftstoffe oder farben , tea , dea , glycol , silikon oder pegs . nur an menschen getested . in anderen worten : wir stellen nur absolut reine produkte her und garantieren mit onezerozero percent sicherheit , dass sie ihrem körper keine chemikalien zuführen .",
         "was es bewirkt das waschgel auf kokosnussbasis entfernt überschüssiges hautfett , während das darin enthaltene aloe vera gel die haut erneuert . das gesichtspflege gel für eine tiefenwirksame porenreinigung . "
         "stimmungsaufhellendes orangenöl für die massage ( kein ätherisches öl für duftlampen ) . ohne paraffin ohne mineralöl , ohne parabene , ohne konservierungsmittel , selbstverständlich ohne tierversuche , vegan",
         "onezerozero percent natives kaltgepresstes biomandelöl aus one . kaltpressung . sanfte und schonende mechanische verarbeitung in deutschland . ",
         "aloe vera , wassernabelkrautextrakt , ackerschachtelhalm extrakt , geranium extract , dandelion extract , natriummethyl two sulfolaurate dinatrium two sulfolaurate , sodium cocoyl isethionat , cocamidopropylbetain , cocamidopropylhydroxysultain , kokosglucoside , natrium chlorid , glyceryl oleat , natriumenzoat , guar hydroxypropyltrimonium chloride , tetrasodium glutamat diacetat , decyl glucoside , sodium levulinate , hydroxamsäure , sodium pca , caprylyl glycol , zitronensäure , als koscher zertifizierte pflanzliches glycerin , eukalyptusöl , pfefferminzöl , zitronengrassöl . zertifiziert als organisch . wir verwenden nur die besten natürlichen zutaten . wenn möglich , verwenden wir onezerozero percent zertifizierte organische zutaten und niemals : petrochemikalien , sulfate , parabene , phthalate oder synthetische duftstoffe oder farben , tea , dea , glycol , silikon oder pegs . nur an menschen getested . in anderen worten : wir stellen nur absolut reine produkte her und garantieren mit onezerozero percent sicherheit , dass sie ihrem körper keine chemikalien zuführen .",
         "was es bewirkt das waschgel auf kokosnussbasis entfernt überschüssiges hautfett , während das darin enthaltene aloe vera gel die haut erneuert . das gesichtspflege gel für eine tiefenwirksame porenreinigung . "
         "stimmungsaufhellendes orangenöl für die massage ( kein ätherisches öl für duftlampen ) . ohne paraffin ohne mineralöl , ohne parabene , ohne konservierungsmittel , selbstverständlich ohne tierversuche , vegan",
         "onezerozero percent natives kaltgepresstes biomandelöl aus one . kaltpressung . sanfte und schonende mechanische verarbeitung in deutschland . ",
         
]

encode = tokenizer(texts, padding='max_length', max_length=200, truncation=True, return_tensors='pt')
for k in encode:
  encode[k] = encode[k].to(device)
```

6. Re-run steps below to see the GPU usage increases every time we run

```python
model.train()
model(**encode)
!nvidia-smi
``` 

Got error as below:

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-242-decae3a1d2bf> in <module>()
      1 model.train()
----> 2 model(**encode)

8 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1674         ret = torch.addmm(bias, input, weight.t())
   1675     else:
-> 1676         output = input.matmul(weight.t())
   1677         if bias is not None:
   1678             output += bias

RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 11.17 GiB total capacity; 10.10 GiB already allocated; 13.81 MiB free; 10.74 GiB reserved in total by PyTorch)

```


However, if you modify the code in the step 6 as follow: 

```
model.train()
output = model(**encode)
print(output)
del output
!nvidia-smi
```
The GPU usage will be stable and the same as every run.


When I'm using batch_size >= 16 with Trainer class I have been facing this issue

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

The GPU usage should stay the same for every run. So that we can run a much bigger batch size.

Right now, I can only use per_device_batch_size <=12 with Trainer class.

Looking forward to learning from you and thank you so much!

<!-- A clear and concise description of what you would expect to happen. -->

Hey @sangnguyen7, 

Is there a reason to rerun these steps:

```python
model.train()
model(**encode)
!nvidia-smi
```
over and over again. I think what might happen here is that pytorch is saving more and more activations of the forward pass in each node of the model and thus will run out of memory eventually. Not sure why you would have do re-run the above steps again and again though.<|||||>Hey @patrickvonplaten, thanks for your response and sorry for late reply. 

Your point might be the case. However, if that is case then it should not be affected by the batch size right? Because if I understood correctly , activation functions are only saved on parameters/weights of the model and they are fixed on each model.
```
Not sure why you would have do re-run the above steps again and again though.
```
The reason why I'm doing this because I want to mimic the training step on the Trainer class to debug which causing the run out of memory... not sure that I missing anything... <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
