Why is the accuracy rate of pre-trained GPT-2 model only ~26%?
Hello,
I have been trying to analyze HellaSwag dataset with the pre-trained GPT2DoubleHeadsModel. I fine-tuned the model by disabling any change fin weights of the main body (12 layers + embedding layer) while training weights of the multiple-choice head with moderate learning rate.

My understanding is that, since the main body of the model is already pre-trained, I should get a reasonably high accuracy rate for the HellaSwag task as long as I do a good job in training the weights from the multiple-choice head. However, the accuracy rate of the pre-trained GPT2DoubleHeadsModel on the HellaSwag task is only ~26% (although my training loss is only ~1.40).

Why is my accuracy rate so low? is this because I am not fine-tuning the weights of the main body of the model during the training? Any advice would be highly appreciated.

Thank you,
Hi, we try to keep the github issues for bugs only. Could you open a thread on the [forum](https://discuss.huggingface.co) instead? Thank you!