CUDA out of memory issue in the middle of training in run_language_modeling.py (say after 1000 steps).
# 🐛 Bug
CUDA OOM in run_language_modeling.py after many steps.
## Information
It seems strange to get them so late in the training procedure.
Model I am using (Bert, XLNet ...):
roberta-large
Language I am using the model on (English, Chinese ...):
English
The problem arises when using:
* [ ] the official example scripts: run_language_modeling.py, with line_by_line parameter

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: Wikitext after some filter of sentences (attached)
[wiki.train.raw.time_filter.normalized.text.txt](https://github.com/huggingface/transformers/files/4191154/wiki.train.raw.time_filter.normalized.text.txt)

## To reproduce
just run the script with the line_by_line parameter
Steps to reproduce the behavior:

- `transformers` version: latest
- Platform:
- Python version: 3.7.0
- PyTorch version (GPU?): latest
- Tensorflow version (GPU?): 
- Using GPU in script?: yes - TITAN Xp
- Using distributed or parallel set-up in script?: no

what is your GPU?<|||||>> what is your GPU?

TITAN Xp
<|||||>If I'm not mistaken the Titan XP has 12GB of VRAM? From my tests training RoBERTa-large with a batch size of 1 already requires 10GB of VRAM, so your GPU memory should be filled quickly. It is surprising that it crashes later though. Could you try with a smaller batch size and gradient accumulation?<|||||>Working with a smaller batch size and gradient accumulation works better. Thank you !