Logits size does not match vocabulary size when using pyctcdecode for a fine-tuned wav2vec 2.0 model
## Environment info
- `transformers` version: 4.17.0.dev0
- Platform: Linux-5.4.0-1063-azure-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.10.2+cu102 (True)
- Tensorflow version (GPU?): 2.6.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes (device = 'cuda')
- Using distributed or parallel set-up in script?: No

## Who can help
@patrickvonplaten, @anton-l

## Dataset used
https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0

## Information
Model I am using: https://huggingface.co/Iskaj/xlsr300m_cv_7.0_nl_lm, (dataset = https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0) The model fine-tuned version of https://huggingface.co/facebook/wav2vec2-xls-r-300m

The problem arises when using:

```python
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F

model_id = "Iskaj/xlsr300m_cv_7.0_nl_lm"

sample_iter = iter(load_dataset("mozilla-foundation/common_voice_8_0", "nl", split="test", streaming=True, use_auth_token=True))

sample = next(sample_iter)
resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()

model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)

input_values = processor(resampled_audio, return_tensors="pt").input_values

with torch.no_grad():
    logits = model(input_values).logits

transcription = processor.batch_decode(logits.numpy()).text
```

## To reproduce

Steps to reproduce the behavior:

1. install packages using: `!pip install https://github.com/kpu/kenlm/archive/master.zip pyctcdecode
!pip install git+https://github.com/huggingface/transformers.git
!pip install git+https://github.com/huggingface/datasets.git
!pip install torchaudio soundfile librosa Levenshtein telwoord wandb jiwer`
2. run: 
```
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F

model_id = "Iskaj/xlsr300m_cv_7.0_nl_lm"

sample_iter = iter(load_dataset("mozilla-foundation/common_voice_8_0", "nl", split="test", streaming=True, use_auth_token=True))

sample = next(sample_iter)
resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()

model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)

input_values = processor(resampled_audio, return_tensors="pt").input_values

with torch.no_grad():
    logits = model(input_values).logits

transcription = processor.batch_decode(logits.numpy()).text
```
3. Observe the error: `ValueError: Input logits of size 48, but vocabulary is size 50

## Expected behavior
I would expect pyctc-decode to work correctly and give me a transcription. 
I suspect it has something to do with `<s>` and `</s>`. I've been struggling a bit with the length of the logits not matching up with the length of the vocabulary, when using pyctcdecode. For example in this repo that uses the LM: https://huggingface.co/patrickvonplaten/wav2vec2-base-100h-with-lm/blob/main/vocab.json the vocab.json includes <s> and </s>, but in this repo it doesn't: https://huggingface.co/hf-test/xls-r-300m-sv/blob/main/vocab.json Maybe that helps
Hey @iskaj,

The problem is that there are 48 tokens in the official vocabulary: https://huggingface.co/Iskaj/xlsr300m_cv_7.0_nl_lm/blob/e0290ad21fd43cf69d9f4de754067c02f9d6641e/vocab.json  , but two more in the alphabet of the language model: https://huggingface.co/Iskaj/xlsr300m_cv_7.0_nl_lm/blob/e0290ad21fd43cf69d9f4de754067c02f9d6641e/alphabet.json

This leads to an error as the two vocabularies have to match.
To fix the problem above we need to do the following change: https://huggingface.co/Iskaj/xlsr300m_cv_7.0_nl_lm/commit/714875260cb8e156ca4e4059bd7d07b863e03aad<|||||>Having done the change, executing your code currently leads to the following error:

```bash
~/python_bin/transformers/models/wav2vec2_with_lm/processing_wav2vec2_with_lm.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    186         missing_decoder_tokens = cls.get_missing_alphabet_tokens(decoder, tokenizer)
    187         if len(missing_decoder_tokens) > 0:
--> 188             raise ValueError(
    189                 f"The tokens {missing_decoder_tokens} are defined in the tokenizer's "
    190                 "vocabulary, but not in the decoder's alphabet. "

ValueError: The tokens {'</s>', '<s>'} are defined in the tokenizer's vocabulary, but not in the decoder's alphabet. Make sure to include {'</s>', '<s>'} in the decoder's alphabet.
```

which is a bug in Transformers that I'll try to fix tomorrow (should be relatively easy) :-) 

I'll keep you posted here<|||||>I indeed tried removing the 2 tokens and encountering the error. Thought because of the error message that it was a fault on my end in terms of modelling. Thanks for the quick response!<|||||>I just took a more detailed look into it and it's actually not really a bug in Wav2Vec2, but a somewhat of an edge case scenario coupled with unintuitive design in Transformers. 

Let's explain. Both your vocabulary file and your alphabet file have been correctly constructed:
- Vocab: https://huggingface.co/Iskaj/xlsr300m_cv_7.0_nl_lm/blob/main/vocab.json
- Alphabet: https://huggingface.co/Iskaj/xlsr300m_cv_7.0_nl_lm/blob/main/alphabet.json
=> They match in vocabulary which is exactly what we want! 

**However**, in Transformers tokenizers can also have additional tokens that when not added to the vocabulary don't necessarily have to have a corresponding output character / logit id. This was the case here. The EOS and BOS token were added as "additional tokens" even though they have no corresponding logit id - see: https://huggingface.co/Iskaj/xlsr300m_cv_7.0_nl_lm/blob/e0290ad21fd43cf69d9f4de754067c02f9d6641e/added_tokens.json . 
This means they correspond to the vocabulary of the tokenizer which in turn would force them to also be part of the alphabet which doesn't make sense though since the alphabet has to correspond 1-to-1 to the logit ids and EOS and BOS have no logit id. 
=> So to make your model work, we actually need to remove those special tokens which I have done in the last three commits:
- https://huggingface.co/Iskaj/xlsr300m_cv_7.0_nl_lm/commit/fa01c8fc60031ec64e9c0cb965ebd2d629e2c4fb
- https://huggingface.co/Iskaj/xlsr300m_cv_7.0_nl_lm/commit/4c37833bd9ac1c717a4dedaaeb328f46160677a1
- https://huggingface.co/Iskaj/xlsr300m_cv_7.0_nl_lm/commit/17f4f68b9c4bd778e087244d882a47eb9c5cdc9a

So the above command now works as expected. 

Finally there is one more difficulty to consider. By default Wav2Vec2CTCTokenizer sets the `eos_token` and the `bos_token` to a value, being `<eos>` and `<bos>` - see: https://huggingface.co/docs/transformers/v4.16.1/en/model_doc/wav2vec2#transformers.Wav2Vec2CTCTokenizer

=> this means we actually need to overwrite this with `None` (or `null` in json). So to remove EOS and BOS one would have to do:

```python
from transformers import Wav2Vec2CTCTokenizer

tok = Wav2Vec2CTCTokenizer.from_pretrained("<path/to/vocab/file/without/eos/and/bos>", eos_token=None, bos_token=None)
```

The original Wav2Vec2 Tokenizers all had EOS and BOS defined in the vocab and a corresponding logit id, which is why it is the default. However this doesn't alwasy have to be the case.<|||||>Hope this help to understand the problem for future use-cases as well :-)<|||||>> Hope this help to understand the problem for future use-cases as well :-)

Patrick, thanks for your help!
Are you planning to fix this issue in Transformers?
Currently this issue makes impossible to run your code example from this repo:
[Transformers Wav2Vec2 + PyCTCDecode](https://github.com/patrickvonplaten/Wav2Vec2_PyCTCDecode)
If vocabulary doesn't contain `<s>, </s>` chars, it will give error described in the thread:

> ValueError: Input logits of size X, but vocabulary is size X + 2<|||||>Hey @BakuDev, 

It's not really an issue in Transformers IMO - one needs to make sure that the tokenizer configs are correct<|||||>> Hey @BakuDev,
> 
> It's not really an issue in Transformers IMO - one needs to make sure that the tokenizer configs are correct

I think I am running into a similar problem.  When I remove/replace` '<s>' and '</s>'` as described above, I am getting the following error, when loading the processor:

```
from transformers import Wav2Vec2ProcessorWithLM, AutoModelForCTC

processor_with_lm = Wav2Vec2ProcessorWithLM.from_pretrained("pgilles/wav2vec2-large-xls-r-LUXEMBOURGISH-with-LM")
```

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-42-3afdbbbb10fb> in <module>
      1 from transformers import Wav2Vec2ProcessorWithLM, AutoModelForCTC
      2 
----> 3 processor_with_lm = Wav2Vec2ProcessorWithLM.from_pretrained("/Users/peter.gilles/Documents/_Daten/Finetuning - Traningsdaten/wav2vec2-large-xls-r-LUXEMBOURGISH2")

/usr/local/lib/python3.8/site-packages/transformers/models/wav2vec2_with_lm/processing_wav2vec2_with_lm.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    186         missing_decoder_tokens = cls.get_missing_alphabet_tokens(decoder, tokenizer)
    187         if len(missing_decoder_tokens) > 0:
--> 188             raise ValueError(
    189                 f"The tokens {missing_decoder_tokens} are defined in the tokenizer's "
    190                 "vocabulary, but not in the decoder's alphabet. "
```

`ValueError: The tokens {'</s>', '<s>'} are defined in the tokenizer's vocabulary, but not in the decoder's alphabet. Make sure to include {'</s>', '<s>'} in the decoder's alphabet.`

When I leave all the `<s>, </s>`, I am getting this error after trying to run an interference:

```
import torch
import torchaudio
import soundfile

waveform, sampling_rate = soundfile.read( audio_sample )
features = processor_with_lm(waveform, sampling_rate=16000, return_tensors="pt")

with torch.no_grad():
  logits = model(**features).logits

transcription = processor_with_lm.batch_decode(logits.numpy()).text
transcription[0].lower()
```

`ValueError: Input logits of size 39, but vocabulary is size 41`

It seems at the moment I am stuck with the integration of a language model.
<|||||>@PeterGilles,

I'm pretty sure the problem is exactly the same as solved in https://github.com/huggingface/transformers/issues/15392#issuecomment-1024905216 . Could you try to solve it following the comments?<|||||>@patrickvonplaten , thanks. I guess my problem is exactly what is described in the comment above. However, I am not succeeding. Maybe I am not getting the last step mentioned in the comment about the Wav2Vec2CTCTokenizer.

My setup at the moment is the following:

```
from pyctcdecode import build_ctcdecoder

decoder = build_ctcdecoder(
    labels=list(sorted_vocab_dict.keys()),
    kenlm_model_path="5gram.arpa",
)

from transformers import Wav2Vec2CTCTokenizer

tokenizer = Wav2Vec2CTCTokenizer.from_pretrained("wav2vec2-large-xls-r-LUXEMBOURGISH-with-LM", eos_token=None, bos_token=None)

from transformers import Wav2Vec2ProcessorWithLM, AutoModelForCTC

processor_with_lm = Wav2Vec2ProcessorWithLM(
    feature_extractor=processor.feature_extractor,
    tokenizer=tokenizer,
    decoder=decoder
)

processor_with_lm.save_pretrained("wav2vec2-large-xls-r-LUXEMBOURGISH-with-LM")

from transformers import Wav2Vec2ProcessorWithLM, AutoModelForCTC

processor_with_lm = Wav2Vec2ProcessorWithLM.from_pretrained("wav2vec2-large-xls-r-LUXEMBOURGISH-with-LM")
model = AutoModelForCTC.from_pretrained("wav2vec2-large-xls-r-LUXEMBOURGISH-with-LM")

import IPython.display as ipd

audio_sample = "pg_8_124_4dfst.wav"
ipd.Audio(data=audio_sample, autoplay=True, rate=audio_sample)

import torch
import torchaudio
import soundfile

waveform, sampling_rate = soundfile.read( audio_sample )

features = processor_with_lm(waveform, sampling_rate=16000, return_tensors="pt")

#input_values = features.input_values
#attention_mask = features.attention_mask

with torch.no_grad():
  logits = model(**features).logits

transcription = processor_with_lm.batch_decode(logits.numpy()).text
transcription[0].lower()
```

`ValueError: Input logits of size 39, but vocabulary is size 41`

The model on [pgilles/wav2vec2-large-xls-r-LUXEMBOURGISH-with-LM](https://huggingface.co/pgilles/wav2vec2-large-xls-r-LUXEMBOURGISH-with-LM) is up-to-date. I think the error comes in when I include the modified tokenizer. 

Thanks.<|||||>Hey @PeterGilles,

I updated your config here: https://huggingface.co/pgilles/wav2vec2-large-xls-r-LUXEMBOURGISH-with-LM/commit/dbfa456977153ef0e9d37f449854de9fddd34595 . The tokenizer should be correct now - can you check your example again? Sorry the LM is too huge for me to check it locally<|||||>Dear @patrickvonplaten , indeed this little change made the difference and it seems to work. Merci!
When running some inferences, the transcription of some audio files which are also part of the training data is running fine. However, when trying with some other audios (2-3 seconds), the output is weird and nonsense, i.e. either one or a few letters or simply nothing. I guess this now a general problem with the language model.<|||||>Hi @patrickvonplaten I got the same error as you mentioned above. I did what you said but I still get an error like below. Can you please help?
ValueError: The tokens {'null'} are defined in the tokenizer's vocabulary, but not in the decoder's alphabet. Make sure to include {'null'} in the decoder's alphabet.

My alphabet.json:
{"labels": ["", "<s>", "</s>", "\u2047", " ", "'", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z", "\u00e7", "\u00f6", "\u00fc", "\u011f", "\u0131", "\u015f"], "is_bpe": false}
my vocab.json:
{"[PAD]": 0, "<s>": 1, "</s>": 2, "[UNK]": 3, "|": 4, "'": 5, "a": 6, "b": 7, "c": 8, "d": 9, "e": 10, "f": 11, "g": 12, "h": 13, "i": 14, "j": 15, "k": 16, "l": 17, "m": 18, "n": 19, "o": 20, "p": 21, "q": 22, "r": 23, "s": 24, "t": 25, "u": 26, "v": 27, "w": 28, "x": 29, "y": 30, "z": 31, "ç": 32, "ö": 33, "ü": 34, "ğ": 35, "ı": 36, "ş": 37}
my added_tokens.json:
{}
my special_tokens_map.json:
{"bos_token": "null", "eos_token": "null", "unk_token": "[UNK]", "pad_token": "[PAD]"}
my tokenizer_config.json:
{"unk_token": "[UNK]", "bos_token": "null", "eos_token": "null", "pad_token": "[PAD]", "do_lower_case": false, "word_delimiter_token": "|", "replace_word_delimiter_char": " ", "special_tokens_map_file": null, "name_or_path": "model/checkpoint-6000", "tokenizer_class": "Wav2Vec2CTCTokenizer", "processor_class": "Wav2Vec2ProcessorWithLM"}


<|||||>I think there should be no " " around null, i.e. `"bos_token": null, "eos_token": null`.<|||||>@patrickvonplaten I have set eos and bos token to null, still i am getting the same error. 
Here is my model
https://huggingface.co/Talha/URDU-ASR<|||||>Hey @talhaanwarch, can you please provide a code snippet showcasing your error?<|||||>@patrickvonplaten  I am looking for the best way to fine-tune wave2vec with trainer of Huggingface and language model ? can you help <|||||>The blog post: https://huggingface.co/blog/fine-tune-xlsr-wav2vec2 should be the most helpful one<|||||>Where does one specify the alpha and beta values for decoding? or is that not applicable here?