Thanks for your reply @sgugger. I have a fine-tuned pretrained model, that is - a AutoModelForCausalLM.from_pretrained("gpt2-large"), which I have fine-tuned on my specific language (which is triage text and quite unlike standard English).
I thought that if I called an AutoModelForSequenceClassification("gpt2-large") then it would be just like reverting to the standard pretrained model, without any of the fine-tuning I have just done, since my fine-tuning resides in the head of the AutoModelForCausalLM.
My question was if it was possible to sit a classifier head on top of the fine tuned language generator model, so as to inherit the weights that I have obtained through the fine-tuning. If I load the weights of the AutoModelForCausalLM into the AutoModelForSequenceClassification then the fine-tuned weights are discarded, but would it be beneficial to add a linear layer to allow those weights to be preserved for use by a manually added classifier head?