Finetuning on several tasks 
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hello, 

Is it possible to finetune a Transformer on some dataset, and then finetune the model again on another dataset with a different number of output labels ? 

I tried this and got the following error: 
```
RuntimeError: Error(s) in loading state_dict for BertForSequenceClassification:
        size mismatch for classifier.weight: copying a param with shape torch.Size([2, 768]) from checkpoint, the shape in current model is torch.Size([3, 768]).
        size mismatch for classifier.bias: copying a param with shape torch.Size([2]) from checkpoint, the shape in current model is torch.Size([3]).
```
Thanks in advance


The `BertForSequenceClassification` model has a classifier head transforming Bert output into `num_labels` output. So if you change the classification output, it can't be loaded as you could see.
The only hack you could do is to load the fine-tuned model with previous `num_labels`, then remove classifier head and replace it with a new classifier with your new `num_labels` (re-initialized) and fine-tune again.
Yet, your Bert model was fine-tuned on a first classification task that is not the same as the second classification task. So if both tasks are too different on too different datasets, it's not sure it will learn anything... or not sure your previous head will give anything decent after that...
<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
