Is the multiple-choice head for the pre-trained `LongformerForMultipleChoice` model pre-trained?
Hello,
Is the multiple-choice head for the pre-trained `LongformerForMultipleChoice` model pre-trained as well?
I am asking because like for the pre-trained `GPT2DoubleHeadsModel`, the main body of the model is trained but its multiple-choice head is not.

Is the multiple-choice head for the pre-trained `LongformerForMultipleChoice` model untrained, just like the case for the `GPT2DoubleHeadsModel`?

Thank you,
Hi! This depends on the checkpoint. If you're using a checkpoint pre-trained on multiple-choice, then it's very possible that it is pre-trained. If you're using a checkpoint pre-trained on another task, it might not be pre-trained.

You should be wary of the task on which the model was trained when leveraging a model with a pre-trained head, as it might not overlap with your current task.