RuntimeError: Error(s) in loading state_dict for Wav2Vec2ForCTC
## Environment info

- `transformers` version: 4.5.0
- Platform: Ubuntu 18.04 
- Python version: 3.7.4

### Who can help

- @patrickvonplaten 

## Information

Model Im using is Wav2vec 2.0.
The problem arises when loading the pretrained/fintuned model:
```
RuntimeError: Error(s) in loading state_dict for Wav2Vec2ForCTC:
	size mismatch for lm_head.weight: copying a param with shape torch.Size([123, 768]) from checkpoint, the shape in current model is torch.Size([132, 768]).
	size mismatch for lm_head.bias: copying a param with shape torch.Size([123]) from checkpoint, the shape in current model is torch.Size([132]).
```

The tasks I am working on is:
* Finetuning Wav2vec 2.0 on my own data from the finetuned xlsr french model.  

## To reproduce

I followed the steps mentioned [here](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2). I started to finetune the base model on my own dataset. Then, I tried to finetune it from the finetuned french xlsr model. 

With fairseq, to fix this problem, we must add the `--restore` argument to say that this model is a finetuned model from a X-architecture.

Any idea about how can we do it using Transformers ?  

Here is the whole error: 

```
RuntimeError: Error(s) in loading state_dict for Wav2Vec2ForCTC:
	size mismatch for lm_head.weight: copying a param with shape torch.Size([123, 768]) from checkpoint, the shape in current model is torch.Size([132, 768]).
	size mismatch for lm_head.bias: copying a param with shape torch.Size([123]) from checkpoint, the shape in current model is torch.Size([132]).
```
Hey @Kamilbentounes,

It looks like your `config.vocab_size` does not match the `config.vocab_size` of the fine-tuned French Wav2Vec2 model. It looks like you want to initialize the model with 132 characters, but the original vocab size is 123. Could you try to align the vocab size to the one of the fine-tuned model? :-) 

If this doesn't fix the problem, please ping me here again<|||||>Hey @patrickvonplaten 

Thanks for your quick reply. Yes, before writing this issue I tried it but I had an error during training phase :/ <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.