Unseen decoded words during inference with Wav2Vec2ProcessorWithLM
## Environment info

- `transformers` version: 4.15.0
- Platform: Linux-5.8.0-36-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


### Who can help
- Wav2Vec2: @patrickvonplaten, @anton-l


## Information
I have been playing with Wav2Vec2ProcessorWithLM mainly with the help of your [helpful blog](https://huggingface.co/blog/wav2vec2-with-ngram) @patrickvonplaten 

In the output of Wav2Vec2ProcessorWithLM, I observed words in the decoded string that were not part of the training text corpus. I am not sure if this is a bug or a feature (OOV detection)? I didn't find any documentation regarding this. So can you verify?
	**Reference**: am i speaking to anushka bohra	
	**Predicted**: am i speaking to anushka **bhra**
The word **bhra** doesn't exist in the arpa LM used.


Hey @sharathadavanne,

Good question and good job investigating the mistake used so far. I assume that you used the LM in combination with a trained acoustic model via `Wav2Vec2ProcessorWithLM` meaning that you used the `pyctcdecode` library: https://github.com/kensho-technologies/pyctcdecode/tree/main/pyctcdecode .

In a nutshell what is happening here is that the predicted word (or letter) is dependent on both the language model (LM) **and** the acoustic model (AM). It's essentially a weighted sum of both the LM and AM's prediction. 

Note that the AM is a CTC model which predicts the next letter. The LM is a n-gram which predicts the next word. `pyctcdecode` takes this into account under the hood an applied a beam search algorithm through the possible transcribed output letters while taking into account the LM. 

Now in case the AM is not very performant or the output just does sound very much like `bhra` , it might give a very high probability to this sequence of letters. The LM will give a probability of close to 0 for this word as it doesn't exist. But since the overall probability is a sum of both AM and LM, it's very well possible that the probability of the AM is so high that `bhra` is decoded.<|||||>Got it. So if I want to make sure all my decoded words are part of my LM, I basically have to use a higher `alpha` in `build_ctcdecoder()`.<|||||>Yes I think this could be a way! BTW - it might be a good idea to actually ask the `pyctcdecode` authors directly on their repo :-)<|||||>BTW, here is also a PR that allows you to change `alpha` at every forward step: https://github.com/huggingface/transformers/pull/15465 . This should make things a bit easier.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.