wav2vec2 with lm on persian doesn't seem to work
### System Info

transformers  version : 4.26.1
Platform: colab
Python version: 3.9.16
PyTorch version (GPU?): 1.13.1+cu116 (False)
Tensorflow version (GPU?): not installed (NA)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using GPU in script?: No (NA)
Using distributed or parallel set-up in script?: No

### Who can help?

@patrickvonplaten 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

i have build a wav2vec2 with lm model followig this blog post [https://huggingface.co/blog/wav2vec2-with-ngram](https://huggingface.co/blog/wav2vec2-with-ngram) 
after building i have tested it on my own data and compared it with wav2vec2 without lm but i can't see any improvement in transcription.

result with lm : 
```

'سالاموروزوخیر خدمتو دوستان عزیز امروز می خوایم با موادی که به صورت '
'رایعج در درمانهای دندانپزشکی استفاده میشه آشنا بشیم خب اولین موادها '
'اسید اچ ها هستن استیج ها در ترم های با کامپوزیت استفاده میشن و به '
'صورت روتین معمولا اسید فسفری که سی و هفت درصد رو استفاده می کنند'
```

result without lm:
```

'سالام موروز وخیر خدمتو دوستان ازیز امروز می خوایم با موادی که به صورت رایعج '
 'در درمان های دندانپزشکی استفاده میشه آشنا بشیم خب اولین مواد ها قسید اج ها '
 'استن اسیلش ها در ترمی هایی با کامپوزیت استفاده میشن و به صورت روتین معمولا '
 'اسید فسفری که سی و هفت درصد رو استفاده می کنن'
```


i also tried to apply fixes mentioned in this [issue](https://github.com/huggingface/transformers/issues/15392) but that also doesn't fix the problem

notebook to reproduce model : [https://colab.research.google.com/drive/1Y_ESjlLd3cbhpmSvpLuR-rWiQTXdd3kq?usp=share_link](https://colab.research.google.com/drive/1Y_ESjlLd3cbhpmSvpLuR-rWiQTXdd3kq?usp=share_link )

i also wanted to push processor to hub but i got some errors so i uploaded files into google drive: 
[https://drive.google.com/drive/folders/1SyEIsd1ZQPBPdDrD97XhdDJs3xhMFNrJ?usp=share_link](https://drive.google.com/drive/folders/1SyEIsd1ZQPBPdDrD97XhdDJs3xhMFNrJ?usp=share_link)


### Expected behavior

at least i expect that lm would fix unigram errors
cc @sanchit-gandhi <|||||>any help?! @sanchit-gandhi <|||||>Hey @tntchack,

Note the adding a LM does not always improve model performance it really depends on what data you train your LM-ngram on. 
While the LM-ngram will not make it impossible for the model to generate spelling errors / unigram errors it should greatly improve spelling errors. If it does not it is a sign that your LM has not been trained very well. 

A couple of tips from my side:
- 1.) Could you try to also contact the Arabic forum to see if someone can help there? https://discuss.huggingface.co/t/arabic-nlp-introductions/3715
- 2.) We have some strong Wav2Vec2-LMs in Arabic on the Hub, e.g. here: https://huggingface.co/kingabzpro/wav2vec2-large-xlsr-300-arabic . Could you try evaluating your model with the LM from this repo instead and see if it improves your perf? You can just load the processor from this repo and the model from your repo<|||||>Hi @patrickvonplaten thanks for your response.

I will have a look on those links and will try Arabic LM with my model<|||||>Hey @patrickvonplaten,
quick questions regarding N-gram:

1. Does N-gram language model for wav2vec2 works upto order of N or sticks to fixed order of N?
2. After the language model is created using the `kenlm` package there's a `unigrams.txt` file inside the directory. Does that files is used by the N-gram model to find appropriate weights while doing the beam search using ctc-decoder?<|||||>$n$-grams typically stay fixed to order $n$ (see https://web.stanford.edu/~jurafsky/slp3/3.pdf). The $n$-gram language model implicitly uses the [pyctcdecode package](https://github.com/kensho-technologies/pyctcdecode) under the hood - the `unigrams.txt` file is used to hold unigram info (see [pyctcdecode/language_model.py#L362-L369](https://github.com/kensho-technologies/pyctcdecode/blob/afecb67622c1395b85b6a55d2902a780c0c927d4/pyctcdecode/language_model.py#L362-L369)).