Finetunig of wav2vec2-xls-r-300m outputs invalid words for Bengali data
### System Info

I have used wav2vec2 pretrained model of wav2vec2-xls-r-300m, and finetuned it to 1000hrs Bengali dataset. Training took 4 full days with 20 epochs. But, there is issue in decoding.  It is decoding in some arbitrary fashion, basically outputs random combination of Bengali letters (which does not have any meaning as confirmed by Bengali natives). It is showing a WER of 100% for all the sentences.

My code is based on the notebook at  https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_Tune_XLS_R_on_Common_Voice.ipynb
@patrickvonplaten, @anton-l @sanchit-gandhi Pls suggest on what could have gone wrong. Should I use fairseq & redo the experiments?
Thanks
Eval loss graph on tensorboard looks like below.
![image](https://user-images.githubusercontent.com/6142443/203315605-d0453531-b0f6-47ad-8e6c-f04c94409938.png)

### Who can help?

@patrickvonplaten, @anton-l, @sanchit-gandhi 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Finetuning on XLSR 300m models using Bengali dataset resulted this behaviour

### Expected behavior

WER should have been less than 100%, and should have outputted reasonably readable hypothesis words
Hey @manjuke! Cool to see you're using XLS-R for fine-tuning on Bengali. The "issues" page on Transformers is reserved for issues related to the Transformers modelling code. For questions related to fine-tuning experiments, the forum is the best place to ask: https://discuss.huggingface.co

Could you copy your question over there? If you tag me (@sanchit-gandhi) I'll aim to answer quickly! Could you also provide a script / colab / command to reproduce this behaviour? From your eval/loss curve, it looks like you're overfitting quite drastically on your training set. This shouldn't happen for a dataset with 1k training hours, so something is definitely up!<|||||>Sure I have created a new ticket @ https://discuss.huggingface.co/t/finetunig-of-wav2vec2-xls-r-300m-outputs-invalid-words-for-bengali-data/26507<|||||>@sanchit-gandhi Could you please reply to my issue raised on hugging face. Awaiting response. Thanks<|||||>Replied on the forum!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.