Would the weights for the main body of the pertained GPT2Model and pertained GPT2DoubleHeadsModel be identical?
Hello,

From my understanding, the difference between the GPT2Model and GPT2DoubleHeadsModel is that GPT2Model does not include any output head, whereas GPT2DoubleHeadsModel includes two types of output heads (lmhead and mchead).

I am wondering, would the weights used in the pertained GPT2Model identical to the weights used in the main body (every parts of the model except the output heads) of the pertained GPT2DoubleHeadsModel? or would the two sets of weights be different since GPT2DoubleHeadsModel was trained after including the output heads, whereas GPT2Model was trained without any output head?

Thank you (I hope my question is understandable),
Hi, except if you freeze the transformer's (transformer being the base model) weights during the training, these will be modified during the fine-tuning. I believe none of our scripts in the library freeze the transformer's weight; so the weights would not be identical.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
