Load pretrained model except the head layer for a specific downstream task
# 🚀 Feature request

It would be nice to have a flag for `from_pretrained` method that indicates whether to load last layer or not. This feature is needed for transfer learning.

## Motivation

I have trained a model with a specific dataset for a downstream task. Now, I need to train another model that needs to be trained on a similar dataset with different labels. I know that previous model have learned the features from the previous dataset and the new model doesn't need to start from scratch. When I try to load the first model with `from_pretrained` method, it returns size mismatch error due to last layer that has different shape for different number of labels. If there is a flag to load/not to load the last layer, I can initialize last layer randomly and go on my training with transfer learning.


for now, how can I load pretrained models that have different prediction heads? thanks!<|||||>For now you can discard the head by passing through a base model:

```py
from transformers import AutoModelForSequenceClassification, AutoModel

pretrained_with_head = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased-distilled-squad")
pretrained_with_head.save_pretrained(directory)
# Model saved in directory has the head

pretrained_no_head = AutoModel.from_pretrained(directory)
pretrained_no_head.save_pretrained(directory)
# Model saved in directory no longer has the head

pretrained_with_head = AutoModelForSequenceClassification.from_pretrained(directory)
# Loaded model has the full transformer, but the head is randomly initialized
```<|||||>Hi @LysandreJik, 
Is this issue being addressed elsewhere? 
If not, would like to work on it. <|||||>@vimarshc this issue has not been addressed elsewhere. Feel free to draft a proposal in an issue/PR so that we can take a look and discuss! Thank you!<|||||>Hi @LysandreJik is this still available for contribution? If yes, I would love to work on it. It would be helpful if you could add a reference draft proposal. Thanks!<|||||>This has been somewhat addressed by https://github.com/huggingface/transformers/pull/12664