GPT-2 finetuning with run_lm_finetuning.py script
## ❓ Questions & Help
I tried to finetune gpt-2 model using `run_lm_finetuning.py` script with the following parameters:
```
python run_lm_finetuning.py \
    --train_data_file=text8.train \
    --output_dir=/content/gpt2 \
    --eval_data_file=text8.val \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --do_eval \
    --per_gpu_train_batch_size=32 \
    --per_gpu_eval_batch_size=32 \
    --gradient_accumulation_steps=1 \
    --num_train_epochs=3 \
    --warmup_steps=200 
``` 
and it throws memory error no matter what my machine type is. In extreme case I wanted to run this demo on Google Cloud with 32 cpus and 120GB RAM - not possible. It just eats the whole RAM and does not even make a single iteration. On the other hand, I was able to do finetuning from this project [gpt-2-simple](https://github.com/minimaxir/gpt-2-simple) on Google Colab with 124M model (CPU). What is going on? Am I doing something wrong?


To me, you have set an extreme `batch_size`. Did you try it with e.g. `per_gpu_train_batch_size=1` and `per_gpu_eval_batch_size=1` ?<|||||>@iedmrc I finally managed to fine-tune it with `per_gpu_train_batch_size=1` and `gradient_accumulation_steps=32`. Indeed the batch size was the problem but I haven't realized it is such a big problem. Everything works fine now.
