Output of finetuned facebook/wav2vec2-xls-r-300m model is getting incorrect 
### System Info

Transformers 4.23.1
Pytorch 1.12.1
Datasets 2.4.0
Tokenizers 0.13.2

I have finetuned  facebook/wav2vec2-xls-r-300m model on my own dataset. I have cross check my dataset twice is it correct format as like libri speech dataset.  I have finetuned my code using reference of @patrickvonplaten
https://huggingface.co/blog/fine-tune-wav2vec2-english

![image](https://user-images.githubusercontent.com/114153974/213086911-a50fab99-6e77-43c8-9320-baa81f23ecc4.png)



### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

a

### Expected behavior

In our case I am getting result like "THTMTHTET"  for IT IS PERMITTED TO TURN THE WHEELPAN BELOW MM WITH THE CONTROLS IN TABLE IT IS PERMITTED TO TURN BELOW THE LAST TURNING' GROOVE

this text we need 
cc @sanchit-gandhi <|||||>Hey @Shubhambugade09,

Could you please try provide a google colab that easily reproduces your training run or maybe instead use the forum specific fine-tuning questions: https://discuss.huggingface.co/ 

Thanks! <|||||>Hey @patrickvonplaten

I have send you the jupyter notebook on [patrick.v.platen@gmail.com](mailto:patrick.v.platen@gmail.com). please check it 

Thanks! <|||||>Hey @Shubhambugade09, would you mind posting a link to your Colab here so I can review it? It would be awesome to post the link here rather than sending as email so that the discussion remains public, this way helping other people who might be experiencing the same issue 🤗 Thanks!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.