Simple LM finetuning falls with RunTime Error: CUDA out of memory
I tried to run simple_lm_finetuning.py on my own data with multi lingual uncased model, and the script breaks down with error 'CUDA out of memory'. Can anyone say what should I do in this situation?
I've already decreased batch size from 32 to 2, but even then I get this error.
![image](https://user-images.githubusercontent.com/37251686/60744362-3ecf9d00-9f7e-11e9-8c3c-7abacce122e6.png)

Offtop: does anybody know, if I can use my own pretrained model in this script instead of one of listed ones?
How much memory does your GPU have. You can check this by running `nvidia-smi`.<|||||>I also has same phenomena. Also, the learning time become slower and much GPU consumption occur, both of which I think is natural, regarding parameters BERT has.

The substitutional way is that, no fine-tuning and dump. 
I mean, feed your sequence to Bert and dump your layers. In the training process of your task, simply load sequence with your dumped result of Bert.<|||||>Try `--reduce_memory `, it worked on mine with base multilingual uncased BERT with batch size= 4 on my single 2080Ti.


```
python3 finetune_on_pregenerated.py 
--pregenerated_data training/ 
--bert_model bert-base-multilingual-uncased 
--do_lower_case 
--output_dir finetuned_lm/ 
--epochs 3 
--reduce_memory 
--train_batch_size 4
```<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
