Change the model type after fine-tuning?
# ❓ Questions & Help

Is there a way in the transformer library to fine-tune a transformer on one downstream task and change to another model type (i.e. another downstream task architecture)?

An example: 
We train a BertForQuestionAnswering model (with a linear layer for the span prediction in addition to the regular BERT). Once we finished our training, I want to disregard the linear layer on top and use the adjusted weights of the 12 BERT-layers in sentiment analysis (BertForSentenceClassification).

I am aware that this would result in a not initialized linear layer on top the BertForSentenceClassification, but that would be not problem in my case. 
Yes, you should be able to just load the finetuned model into another architecture. The weights of overlapping parameter names will be loaded, and the others will be ignored.<|||||>Awesome, thanks for the fast response <|||||>Okay,  I came across a followup question: 
So from what I can tell, the pre-trained models come with the weights for the BertForMaskedLM (as this has been used to do the pre-training).
Is there a way to first fine-tune a model on a downstream task (like sentence classification) and than reload it as BertForMaskedLM, reusing the trained LM head?   <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>@BramVanroy could you provide a code example of how to this? I want to do sequential training on different tasks without having to save and load models from disk, or change layers directly myself. Ideally I would use some head switcher function or simply have `.from_pretrained()` accept model instances.<|||||>@timellemeet Probably best to post a new issue or [a forum post](https://discuss.huggingface.co/).