PyTorch (with GPU) Trainer leaks CPU memory on Google Colab
## Environment info
- `transformers` version: 3.1.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


### Who can help
Trainer: @sgugger

## Information

Model I am using `RoBERTaForMaskedLM`:
`
config = RobertaConfig(
    vocab_size=32_000,
    max_position_embeddings=256+2,
    num_attention_heads=12,
    num_hidden_layers=6,
    type_vocab_size=1,
)
`
The problem arises when using:
* [ ] [my own modified scripts](https://colab.research.google.com/drive/1bVv6V9IIhTNXvWbpUvQyPyRpK3LOgFfz?usp=sharing): I am trying to follow the official blog tutorial on training a language model from scratch. I have made a few changes from the official script (using Marathi OSCAR corpus, changed model config, and vocabulary size, and am fetching the sentences in the dataset on the fly

The tasks I am working on is:
* [ ] my own task or dataset: Masked Language Modelling with RoBERTa on Marathi Oscar Corpus

## To reproduce

Steps to reproduce the behavior:

1. Run this [colab notebook](https://colab.research.google.com/drive/1bVv6V9IIhTNXvWbpUvQyPyRpK3LOgFfz?usp=sharing).

## Expected behavior
The RAM consumption starts rising towards the end of the first epoch, ultimately crashing the entire session due to full memory consumption (12.72GB of Colab RAM)
