ValueError: The tokens {'null'} are defined in the tokenizer's vocabulary, but not in the decoder's alphabet. Make sure to include {'null'} in the decoder's alphabet.
### System Info

```shell
Hi @patrickvonplaten I got the same error as you mentioned above. I did what you said but I still get an error like below. Can you please help?
ValueError: The tokens {'null'} are defined in the tokenizer's vocabulary, but not in the decoder's alphabet. Make sure to include {'null'} in the decoder's alphabet.

My alphabet.json:
{"labels": ["", "", "", "\u2047", " ", "'", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z", "\u00e7", "\u00f6", "\u00fc", "\u011f", "\u0131", "\u015f"], "is_bpe": false}
my vocab.json:
{"[PAD]": 0, "": 1, "": 2, "[UNK]": 3, "|": 4, "'": 5, "a": 6, "b": 7, "c": 8, "d": 9, "e": 10, "f": 11, "g": 12, "h": 13, "i": 14, "j": 15, "k": 16, "l": 17, "m": 18, "n": 19, "o": 20, "p": 21, "q": 22, "r": 23, "s": 24, "t": 25, "u": 26, "v": 27, "w": 28, "x": 29, "y": 30, "z": 31, "ç": 32, "ö": 33, "ü": 34, "ğ": 35, "ı": 36, "ş": 37}
my added_tokens.json:
{}
my special_tokens_map.json:
{"bos_token": "null", "eos_token": "null", "unk_token": "[UNK]", "pad_token": "[PAD]"}
my tokenizer_config.json:
{"unk_token": "[UNK]", "bos_token": "null", "eos_token": "null", "pad_token": "[PAD]", "do_lower_case": false, "word_delimiter_token": "|", "replace_word_delimiter_char": " ", "special_tokens_map_file": null, "name_or_path": "model/checkpoint-6000", "tokenizer_class": "Wav2Vec2CTCTokenizer", "processor_class": "Wav2Vec2ProcessorWithLM"}
```


### Who can help?

@patrickvonplaten 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Hi @patrickvonplaten I got the same error as you mentioned above. I did what you said but I still get an error like below. Can you please help?
ValueError: The tokens {'null'} are defined in the tokenizer's vocabulary, but not in the decoder's alphabet. Make sure to include {'null'} in the decoder's alphabet.

My alphabet.json:
{"labels": ["", "", "", "\u2047", " ", "'", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z", "\u00e7", "\u00f6", "\u00fc", "\u011f", "\u0131", "\u015f"], "is_bpe": false}
my vocab.json:
{"[PAD]": 0, "": 1, "": 2, "[UNK]": 3, "|": 4, "'": 5, "a": 6, "b": 7, "c": 8, "d": 9, "e": 10, "f": 11, "g": 12, "h": 13, "i": 14, "j": 15, "k": 16, "l": 17, "m": 18, "n": 19, "o": 20, "p": 21, "q": 22, "r": 23, "s": 24, "t": 25, "u": 26, "v": 27, "w": 28, "x": 29, "y": 30, "z": 31, "ç": 32, "ö": 33, "ü": 34, "ğ": 35, "ı": 36, "ş": 37}
my added_tokens.json:
{}
my special_tokens_map.json:
{"bos_token": "null", "eos_token": "null", "unk_token": "[UNK]", "pad_token": "[PAD]"}
my tokenizer_config.json:
{"unk_token": "[UNK]", "bos_token": "null", "eos_token": "null", "pad_token": "[PAD]", "do_lower_case": false, "word_delimiter_token": "|", "replace_word_delimiter_char": " ", "special_tokens_map_file": null, "name_or_path": "model/checkpoint-6000", "tokenizer_class": "Wav2Vec2CTCTokenizer", "processor_class": "Wav2Vec2ProcessorWithLM"}

### Expected behavior

```shell
Hi @patrickvonplaten I got the same error as you mentioned above. I did what you said but I still get an error like below. Can you please help?
ValueError: The tokens {'null'} are defined in the tokenizer's vocabulary, but not in the decoder's alphabet. Make sure to include {'null'} in the decoder's alphabet.

My alphabet.json:
{"labels": ["", "", "", "\u2047", " ", "'", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z", "\u00e7", "\u00f6", "\u00fc", "\u011f", "\u0131", "\u015f"], "is_bpe": false}
my vocab.json:
{"[PAD]": 0, "": 1, "": 2, "[UNK]": 3, "|": 4, "'": 5, "a": 6, "b": 7, "c": 8, "d": 9, "e": 10, "f": 11, "g": 12, "h": 13, "i": 14, "j": 15, "k": 16, "l": 17, "m": 18, "n": 19, "o": 20, "p": 21, "q": 22, "r": 23, "s": 24, "t": 25, "u": 26, "v": 27, "w": 28, "x": 29, "y": 30, "z": 31, "ç": 32, "ö": 33, "ü": 34, "ğ": 35, "ı": 36, "ş": 37}
my added_tokens.json:
{}
my special_tokens_map.json:
{"bos_token": "null", "eos_token": "null", "unk_token": "[UNK]", "pad_token": "[PAD]"}
my tokenizer_config.json:
{"unk_token": "[UNK]", "bos_token": "null", "eos_token": "null", "pad_token": "[PAD]", "do_lower_case": false, "word_delimiter_token": "|", "replace_word_delimiter_char": " ", "special_tokens_map_file": null, "name_or_path": "model/checkpoint-6000", "tokenizer_class": "Wav2Vec2CTCTokenizer", "processor_class": "Wav2Vec2ProcessorWithLM"}
```

Hey @erdoganensar,

Could you provide a code snippet that shows how I can reproduce the error?<|||||>I have a very similar problem related to this issue using HuBERT large on English corpus. Here is my code snippet:

```
processor = Wav2Vec2Processor.from_pretrained("facebook/hubert-large-ls960-ft")
vocab_dict = processor.tokenizer.get_vocab()
sorted_vocab_dict = {k.lower(): v for k, v in sorted(vocab_dict.items(), key=lambda item: item[1])}

decoder = build_ctcdecoder(
    labels=list(sorted_vocab_dict.keys()),
    kenlm_model_path="some_3gram_correct.arpa",
)
processor_with_lm = Wav2Vec2ProcessorWithLM(
    feature_extractor=processor.feature_extractor,
    tokenizer=processor.tokenizer,
    decoder=decoder
)
```

Here is the first 20 lines of the `some_3gram_correct.arpa`:

```
\data\
ngram 1=759
ngram 2=3580
ngram 3=5747

\1-grams:
-0.77224493	<unk>
-inf	<s>	-0.96890455
-inf	</s>	-0.96890455
-1.0275165	</s>
-1.8815907	it	-0.576264
-2.4739406	looks	-0.47474432
-2.598626	like	-0.19285807
-1.7276717	a	-0.42839557
-2.9495435	nice	-0.13558547
-2.9495435	day	-0.3788458
-2.122181	outside	-0.6023683
-1.761344	that	-0.4610282
-2.0622265	's	-0.32115284
-2.6496518	about	-0.41595408
```

Here is the error message I got:

`ValueError: The tokens {'H', 'Y', 'Q', 'M', 'D', 'I', 'F', 'P', 'J', 'V', 'X', 'B', 'C', 'U', 'E', 'S', 'N', 'R', 'Z', 'L', 'T', 'K', 'A', 'G', 'O', 'W'} are defined in the tokenizer's vocabulary, but not in the decoder's alphabet. Make sure to include {'H', 'Y', 'Q', 'M', 'D', 'I', 'F', 'P', 'J', 'V', 'X', 'B', 'C', 'U', 'E', 'S', 'N', 'R', 'Z', 'L', 'T', 'K', 'A', 'G', 'O', 'W'} in the decoder's alphabet.`

How should proceed? Thanks in advance.<|||||>@erdoganensar note that the problem here is that the tokenizer's vocab has upper-case letters but the decoder has lowercase letters. Now from your 3gram it looks like the decoder should indeed have lowercase letters. So what you should do here is the following before running the above code snippet:

```python
processor = Wav2Vec2Processor.from_pretrained("facebook/hubert-large-ls960-ft")
tokenizer_vocab_dict = processor.tokenizer.get_vocab()
tokenizer_vocab_lowercase = {k.lower(): v for k,v in tokenizer_vocab_dict.items()}

vocab_file = "vocab.json"
with open(vocab_file, "w", encoding="utf-8") as f: 
    f.write(json.dumps(tokenizer_vocab_lowercase, ensure_ascii=False))

processor.tokenizer = Wav2Vec2CTCTokenizer(vocab_file)
processor.save_pretrained("path/to/processor")
```

Having done this you can execute the following code which should then work correctly:

```python
processor = Wav2Vec2Processor.from_pretrained("path/to/processor")
vocab_dict = processor.tokenizer.get_vocab()
sorted_vocab_dict = {k.lower(): v for k, v in sorted(vocab_dict.items(), key=lambda item: item[1])}

decoder = build_ctcdecoder(
    labels=list(sorted_vocab_dict.keys()),
    kenlm_model_path="some_3gram_correct.arpa",
)
processor_with_lm = Wav2Vec2ProcessorWithLM(
    feature_extractor=processor.feature_extractor,
    tokenizer=processor.tokenizer,
    decoder=decoder
)
```<|||||>@patrickvonplaten Thanks!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.