RuntimeError: CUDA out of memory. Tried to allocate 786.00 MiB (GPU 0; 14.73 GiB total capacity; 13.33 GiB already allocated; 575.88 MiB free; 13.38 GiB reserved in total by PyTorch)
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT2
Language I am using the model on (English, Chinese ...): eng

**The tasks I am working on is:**

!python /content/transformers/examples/run_language_modeling.py --train_data_file=shakespeare.txt --model_type=gpt2 --model_name_or_path=gpt2 --output_dir=output --do_train

## To reproduce

Steps to reproduce the behavior:

import os
import requests
file_name = "shakespeare.txt"
if not os.path.isfile(file_name):
	url = "https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt"
	data = requests.get(url)
	
	with open(file_name, 'w') as f:
		f.write(data.text)
!python /content/transformers/examples/run_language_modeling.py --train_data_file=shakespeare.txt --model_type=gpt2 --model_name_or_path=gpt2 --output_dir=output --do_train



03/20/2020 13:36:52 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: False
03/20/2020 13:36:53 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json from cache at /root/.cache/torch/transformers/4be02c5697d91738003fb1685c9872f284166aa32e061576bbe6aaeb95649fcf.699bbd1c449e9861456f359d6daa51bd523ac085b4b531ab0aad5a55d091e942
03/20/2020 13:36:53 - INFO - transformers.configuration_utils -   Model config GPT2Config {
  "architectures": [
    "GPT2LMHeadModel"
  ],
  "attn_pdrop": 0.1,
  "bos_token_id": null,
  "do_sample": false,
  "embd_pdrop": 0.1,
  "eos_token_ids": null,
  "finetuning_task": null,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "layer_norm_epsilon": 1e-05,
  "length_penalty": 1.0,
  "max_length": 20,
  "model_type": "gpt2",
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12,
  "n_positions": 1024,
  "num_beams": 1,
  "num_labels": 2,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_token_id": null,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 50257
}

03/20/2020 13:36:54 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json from cache at /root/.cache/torch/transformers/f2808208f9bec2320371a9f5f891c184ae0b674ef866b79c58177067d15732dd.1512018be4ba4e8726e41b9145129dc30651ea4fec86aa61f4b9f40bf94eac71
03/20/2020 13:36:54 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt from cache at /root/.cache/torch/transformers/d629f792e430b3c76a1291bb2766b0a047e36fae0588f9dbc1ae51decdff691b.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda
03/20/2020 13:36:54 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-pytorch_model.bin from cache at /root/.cache/torch/transformers/4295d67f022061768f4adc386234dbdb781c814c39662dd1662221c309962c55.778cf36f5c4e5d94c8cd9cefcf2a580c8643570eb327f0d4a1f007fab2acbdf1
03/20/2020 13:37:02 - INFO - __main__ -   Training/evaluation parameters Namespace(adam_epsilon=1e-08, block_size=1024, cache_dir=None, config_name=None, device=device(type='cuda'), do_eval=False, do_train=True, eval_all_checkpoints=False, eval_data_file=None, evaluate_during_training=False, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, learning_rate=5e-05, line_by_line=False, local_rank=-1, logging_steps=500, max_grad_norm=1.0, max_steps=-1, mlm=False, mlm_probability=0.15, model_name_or_path='gpt2', model_type='gpt2', n_gpu=1, no_cuda=False, num_train_epochs=1.0, output_dir='output', overwrite_cache=False, overwrite_output_dir=False, per_gpu_eval_batch_size=4, per_gpu_train_batch_size=4, save_steps=500, save_total_limit=None, seed=42, server_ip='', server_port='', should_continue=False, tokenizer_name=None, train_data_file='shakespeare.txt', warmup_steps=0, weight_decay=0.0)
03/20/2020 13:37:02 - INFO - __main__ -   Loading features from cached file gpt2_cached_lm_1024_shakespeare.txt
03/20/2020 13:37:02 - INFO - __main__ -   ***** Running training *****
03/20/2020 13:37:02 - INFO - __main__ -     Num examples = 330
03/20/2020 13:37:02 - INFO - __main__ -     Num Epochs = 1
03/20/2020 13:37:02 - INFO - __main__ -     Instantaneous batch size per GPU = 4
03/20/2020 13:37:02 - INFO - __main__ -     Total train batch size (w. parallel, distributed & accumulation) = 4
03/20/2020 13:37:02 - INFO - __main__ -     Gradient Accumulation steps = 1
03/20/2020 13:37:02 - INFO - __main__ -     Total optimization steps = 83
Epoch:   0% 0/1 [00:00<?, ?it/s]
Iteration:   0% 0/83 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/content/transformers/examples/run_language_modeling.py", line 799, in <module>
    main()
  File "/content/transformers/examples/run_language_modeling.py", line 749, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "/content/transformers/examples/run_language_modeling.py", line 353, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 602, in forward
    shift_logits = lm_logits[..., :-1, :].contiguous()
RuntimeError: CUDA out of memory. Tried to allocate 786.00 MiB (GPU 0; 14.73 GiB total capacity; 13.33 GiB already allocated; 575.88 MiB free; 13.38 GiB reserved in total by PyTorch)

Epoch:   0% 0/1 [00:00<?, ?it/s]
Iteration:   0% 0/83 [00:00<?, ?it/s]



@mariamabarham didn't you have a similar issue? <|||||>Yes I encountered the same issue. I solved it by adding --fp16(need to install apex first). You can also reduce the block_size to 512. Both worked out for me.<|||||>You should probably set the `per_gpu_train_batch_size` to 1. That is the default behavior for `gpt-2-simple` to prevent OOM. (I am not a fan of the default batch_size of 4 in `run_language_modeling.py`)<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>You can also try using gradient accumulation steps. 
Basically, if you want a batch_size of 32, but your GPU can only fit 16.

So you make two passes of 16 batches each, accumulate your gradients, and then do the backward pass after 2 batches.
