Conflict between pyctcdecode and Wav2Vec2ProcessorWithLM
### System Info

transformers 4975002df50c472cbb6f8ac3580e475f570606ab
pyctcdecode 9afead58560df07c021aa01285cd941f70fe93d5



### Who can help?

@patrici

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Error:
` The tokens {'', '⁇', ' '} are defined in the tokenizer's vocabulary, but not in the decoder's alphabet. Make sure to include {'', '⁇', ' '} in the decoder's alphabet.`

Reason:
`get_missing_alphabet_tokens` will replace special tokens

https://github.com/huggingface/transformers/blob/4975002df50c472cbb6f8ac3580e475f570606ab/src/transformers/models/wav2vec2_with_lm/processing_wav2vec2_with_lm.py#L196

however if we `build_ctcdecoder` using the same tokenizer vocab, it will be always mismatch.


### Expected behavior

A straight fix is do the same mapping on build_ctcdecoder

```python
from transformers import AutoProcessor
from pyctcdecode.alphabet import BLANK_TOKEN_PTN, UNK_TOKEN, UNK_TOKEN_PTN, Alphabet
from pyctcdecode import build_ctcdecoder
from transformers import Wav2Vec2ProcessorWithLM

model_to_add_lm = "wav2vec2-large-xxxxx"
lm_arpa_path = "xxxxx.arpa"

processor = AutoProcessor.from_pretrained(model_to_add_lm)
vocab_dict = processor.tokenizer.get_vocab()
sorted_vocab_dict = {k: v for k, v in sorted(vocab_dict.items(), key=lambda item: item[1])}
alphabet = list(sorted_vocab_dict.keys())

for i, token in enumerate(alphabet):
    if BLANK_TOKEN_PTN.match(token):
        alphabet[i] = ""
    if token == processor.tokenizer.word_delimiter_token:
        alphabet[i] = " "
    if UNK_TOKEN_PTN.match(token):
        alphabet[i] = UNK_TOKEN

decoder = build_ctcdecoder(
    labels=alphabet,
    kenlm_model_path=lm_arpa_path,
)

decoder._alphabet._labels = alphabet
processor_with_lm = Wav2Vec2ProcessorWithLM(
    feature_extractor=processor.feature_extractor,
    tokenizer=processor.tokenizer,
    decoder=decoder
)

processor_with_lm.save_pretrained("xxxxxx")

```
Maybe of interest to @patrickvonplaten @anton-l @sanchit-gandhi** <|||||>Hi @voidful. The function [`get_missing_alphabet_tokens`](https://github.com/huggingface/transformers/blob/99eb9b523f9b9ea6096323ce5610ce6633acc88a/src/transformers/models/wav2vec2_with_lm/processing_wav2vec2_with_lm.py#L187) will only replace 'special' tokens associated with CTC decoding, namely:

- The 'blank' token
- The 'pad' token
- The 'word delimiter' token

The function is used to highlight discrepancies between the tokenizer and decoder vocabularies. The tokens highlighted as missing `{'', '⁇', ' '}` are not filtered by this function, and thus appear to be missing in the decoder vocabulary.

May I ask, what is it exactly that you are proposing? If we you could provide a code snippet to reproduce this behaviour it would be much appreciated.<|||||>My situation is that I have a fine-tuned xlsr model, and I want to add kenlm on top of it. 
And I build the decoder using `build_ctcdecoder`, the label will be the same as our tokenizer vocabulary.
Therefore It will have discrepancies  on https://github.com/huggingface/transformers/blob/99eb9b523f9b9ea6096323ce5610ce6633acc88a/src/transformers/models/wav2vec2_with_lm/processing_wav2vec2_with_lm.py#L187

I suggest not to replace the token when  tokenizer and decoder vocabularies are the same.

Here is my code:
https://colab.research.google.com/drive/1IR8cwVjkflJhj0e7te_iAdYfuNlKDVzr?usp=sharing
<|||||>Thanks for the code-snippet! I haven't been able to reproduce on the other template examples (e.g. https://discuss.huggingface.co/t/how-to-create-wav2vec2-with-language-model/12703). Will look more in-depth as to why the exception is being thrown for the use case in the Colab!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Hey @voidful, sorry about the delayed reply. I've taken a deeper look into your issue - it looks as though there is a mis-match between the tokeniser and LM's vocabularies (12305 tokens to be exact): https://colab.research.google.com/drive/1v1qd4CUdSXKmrSYIMqMzMk_KCUMfMWu9?usp=sharing

For LM boosted beam-search decoding for CTC, we need the vocabulary of the LM to match that of the tokeniser one-to-one. You can ensure this by training your LM using the same method that you use to train the Wav2Vec2 tokeniser. You then shouldn't have to override the method `decoder._alphabel.labels`: the vocabularies should already match (barring the special tokens).

See this example for creating a tokeniser: https://github.com/sanchit-gandhi/seq2seq-speech/blob/main/get_ctc_tokenizer.py

And this example for creating a corresponding LM: https://github.com/sanchit-gandhi/seq2seq-speech/blob/main/get_ctc_ngram.py

This blog also explains succinctly how one can train and instantiate an LM: https://huggingface.co/blog/wav2vec2-with-ngram<|||||>> Hey @voidful, sorry about the delayed reply. I've taken a deeper look into your issue - it looks as though there is a mis-match between the tokeniser and LM's vocabularies (12305 tokens to be exact): https://colab.research.google.com/drive/1v1qd4CUdSXKmrSYIMqMzMk_KCUMfMWu9?usp=sharing
> 
> For LM boosted beam-search decoding for CTC, we need the vocabulary of the LM to match that of the tokeniser one-to-one. You can ensure this by training your LM using the same method that you use to train the Wav2Vec2 tokeniser. You then shouldn't have to override the method `decoder._alphabel.labels`: the vocabularies should already match (barring the special tokens).
> 
> See this example for creating a tokeniser: https://github.com/sanchit-gandhi/seq2seq-speech/blob/main/get_ctc_tokenizer.py
> 
> And this example for creating a corresponding LM: https://github.com/sanchit-gandhi/seq2seq-speech/blob/main/get_ctc_ngram.py
> 
> This blog also explains succinctly how one can train and instantiate an LM: https://huggingface.co/blog/wav2vec2-with-ngram

I see, the reason is that I use a bpe vocabulary to train the ctc model, it will not be match to KenLM, so I have to patch the vocabulary to make sure not deleting the bpe token.  