Pretrained GPT2DoubleHeadsModel
#6667 
Hello,
This is a follow-up post for #6667.

So just so that I am understanding this correctly, the main body (excluding the 2 output heads) of the pre-trained `GPT2DoubleHeadsModel`  are indeed the pre-trained weights and biases, but the 2 output heads are not pre-trained?

Thank you,
One of the two output heads is the language modeling head, which is tied to the embeddings. This is already trained, as the embeddings were trained during pre-training.

The second output head is a multiple choice head, which was not pre-trained. You would need to fine-tune it on a multiple choice dataset so that it works in your case.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
