OpenAIGPTDoubleHeadsModel throws CUDA OOM with large number of candidates
# 🐛 Bug

I am trying to train the `OpenAIGPTDoubleHeadsModel`. I find that large number of candidates can cause CUDA OOM errors.

Case 1 (single training example with 67 candidates): CUDA OOM
```
input_ids.shape: torch.Size([1, 67, 275])
mc_token_ids.shape: torch.Size([1, 67])
lm_labels.shape: torch.Size([1, 67, 275])
mc_labels.shape: torch.Size([1])
token_type_ids.shape: torch.Size([1, 67, 275])
```

Case 2 (single training example with 3 candidates): works fine!
```
input_ids.shape: torch.Size([1, 3, 275])
mc_token_ids.shape: torch.Size([1, 3])
lm_labels.shape: torch.Size([1, 3, 275])
mc_labels.shape: torch.Size([1])
token_type_ids.shape: torch.Size([1, 3, 275])
```

## Information

Model I am using: `OpenAIGPTDoubleHeadsModel`

Language I am using the model on: English

The problem arises when using my own modified scripts based on the `transfer-learning-conv-ai` repo by Hugging Face.

## To reproduce
Simply try training `OpenAIGPTDoubleHeadsModel` with larger number of candidates (such as 67).

## Expected behavior

3 or 67 candidates shouldn't matter, both cases 1 and 2 should work fine without CUDA OOM.

## Environment info
     
- `transformers` version: 2.3.0
- Platform: Amazon Linux (Deep Learning AMI)
- Python version: 3.6
- PyTorch version (GPU?): the one shipped with the pytorch_p36 conda env in Amazon DL AMI
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes and No

Isn't that to be expected? The large model just doesn't fit on your GPU's memory.<|||||>@BramVanroy is that expected behavior with just 1 training example though?

I initially suspected that this is a training-specific behavior (due to the need to store gradients, etc.), so I decided to fix the number of candidates to something small during training.

I then attempted to do inference with this trained model, but I used all candidates during inference. I took 1 example to do inference on, and I observed memory issues with that too.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
