run_generation.py GPT2 model only using 1st GPU, OOM error
# 🐛 Bug

## Information

GPT2-xl (pre-trained)

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x ] my own modified scripts: (give details below)
Run in jupyter notebok script below, GCP T4x2

## To reproduce

Steps to reproduce the behavior:
```
prompt_text="If God is defined as something that is all powerful and all knowing, a strong artificial intelligence might be an actual God. If this happens the implications for religion are"
num_of_responses=15
!python transformers/examples/text-generation/run_generation.py --model_type GPT2 --model_name_or_path /spell/GPT2Model/GPT2Model/ --length 1000 --stop_token '<|endoftext|>' --temperature .7 --repetition_penalty 85 --prompt "$prompt_text" --num_return_sequences $num_of_responses
```
Nvidia-smi during run only reports gpu1 using memory  gpu2 sits at 11m and never moves
Ultimately program errors out with OOM error:

Traceback (most recent call last):
  File "transformers/examples/text-generation/run_generation.py", line 268, in <module>
    main()
  File "transformers/examples/text-generation/run_generation.py", line 237, in main
    num_return_sequences=args.num_return_sequences,
  File "/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py", line 1148, in generate
    model_specific_kwargs=model_specific_kwargs,
  File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py", line 1190, in _generate_no_beam_search
    outputs = self(**model_inputs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_gpt2.py", line 615, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_gpt2.py", line 499, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_gpt2.py", line 236, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_gpt2.py", line 191, in forward
    present = torch.stack((key.transpose(-2, -1), value))  # transpose to have same shapes for stacking
RuntimeError: CUDA out of memory. Tried to allocate 84.00 MiB (GPU 0; 14.76 GiB total capacity; 13.69 GiB already allocated; 37.44 MiB free; 13.93 GiB reserved in total by PyTorch)
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- `transformers` version: 2.9.0
- Platform: Linux-5.3.0-1018-gcp-x86_64-with-glibc2.10
- Python version: 3.8.2
- PyTorch version (GPU?): 1.5.0+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: 
- Using distributed or parallel set-up in script?: 
I don't think we currently support multi-gpu in this script.<|||||>16 GB VRAM might not be enough to allow batching of generated texts.<|||||>Actually, if you install apex and are able to cast the model as FP16 using `model.half()`, you can get a batch size of up to 30 for the 1.5B. (nvidia-smi is reporting 14GB VRAM used on a single T4)

My tool is slightly different than the script, but I'll be sure to include bulk 1.5B generation as a documentation example.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
