How to solve the CUDA out of memory
ERROR:


> 07/19/2021 10:04:35 - INFO - __main__ - ***** Running training *****
> 07/19/2021 10:04:35 - INFO - __main__ -   Num examples = 23
> 07/19/2021 10:04:35 - INFO - __main__ -   Num Epochs = 7██████| 1/1 [00:00<00:00,  4.48ba/s]
> 07/19/2021 10:04:35 - INFO - __main__ -   Instantaneous batch size per device = 8  3.76ba/s]
> 07/19/2021 10:04:35 - INFO - __main__ -   Total train batch size (w. parallel, distributed & accumulation) = 8
> 07/19/2021 10:04:35 - INFO - __main__ -   Gradient Accumulation steps = 1
> 07/19/2021 10:04:35 - INFO - __main__ -   Total optimization steps = 21
>   0%|                                                                | 0/21 [00:00<?, ?it/s]Traceback (most recent call last):
>   File "./run_clm_no_trainer.py", line 496, in <module>
>     main()
>   File "./run_clm_no_trainer.py", line 460, in main
>     accelerator.backward(loss)
>   File "/usr/local/lib/python3.6/dist-packages/accelerate/accelerator.py", line 251, in backward
>     loss.backward()
>   File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 245, in backward
>     torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
>   File "/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py", line 147, in backward
>     allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
> RuntimeError: CUDA out of memory. Tried to allocate 1.53 GiB (GPU 0; 31.75 GiB total capacity; 30.16 GiB already allocated; 221.75 MiB free; 30.22 GiB reserved in total by PyTorch)
>   0%|                                                                | 0/21 [00:00<?, ?it/s]



I use the official example of finetuning gpt2 from https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_clm_no_trainer.py
No modifications on this code.

```
CUDA_VISIBLE_DEVICES=1 python ./run_clm_no_trainer.py \
--num_train_epochs 7 \
--model_name_or_path gpt2 \
--per_device_train_batch_size 8 \
--per_device_eval_batch_size 8 \
--output_dir ./finetune_cache \
--preprocessing_num_workers 8 
```

My GPU device 1 is of 32510M CUDA memory, V100.
How can I solve the memory error, without degrading the performance of the finetuned gpt2 ?
I changed the param of block_size to 512, it solves the memory error. Will this change have effects on the performance of gpt2 ?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.