Confused by GPT2DoubleHeadsModel example
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Your library is really helpful. And I have 2 questions about the example of GPT2DoubleHeadsModel.

1. 
In source code of model, 

> choices = ["Hello, my dog is cute [CLS]", "Hello, my cat is cute [CLS]"]

Here the comment said that "number of choices: 2". So is it a single sample or two samples? What are the two label classes?

2.
What task is the GPT2DOUBLEHeadsModel trained on? I don't find the information from the document, issues or the original paper.

Please help me. Thank you in advance. 
For 2: It's next word prediction: https://openai.com/blog/better-language-models/<|||||>> For 2: It's next word prediction: https://openai.com/blog/better-language-models/

Thank you for your reply. From my perspective, I believe the next word prediction is what the language model does, which means it is the "unsupervised" part in the first head. 

The evidence is that gpt-2 is based on gpt. Based on figure 1 in gpt paper, the left head is for "text prediction"(next word prediction), and the right head is for "task prediction".<|||||>Hi, the GPT2DoubleHeadsModel, as defined in [the documentation](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2doubleheadsmodel), is: "_The GPT2 Model transformer with a language modeling and a multiple-choice classification head on top e.g. for RocStories/SWAG tasks. The two heads are two linear layers. The language modeling head has its weights tied to the input embeddings, the classification head takes as input the input of a specified classification token index in the input sequence)._".

You use a double head model when you want to have two separate losses. In this case the two linear layers compute two different losses: one of the heads is a language modeling loss, with a linear layer of size `hidden_size` x `vocab_size` and the other is a classification loss with a linear layer of size `hidden size` x `number_of_choices`.

You can read [this blog post](https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313) by @thomwolf which uses GPT2DoubleHeadsModel with a previous version of this library (pytorch-pretrained-BERT).<|||||>@LysandreJik 
Hi, thank you for your help!