Found the issue. While building vocabulary we replace the blank token " " by pipe symbol as delimiter ("|"). But if the dataset already contains the pipe symbol, which is common in Hindi language as Hindi language uses a similar character “।” as a full stop, in this case we will get two pipe symbols in the vocabulary. This will mess up the further process of creating the dictionary which is adding special tokens. Specifically, the bos token “<s>” and the pad token “[PAD]” will get the same index in vocabulary. This will cause the index with [PAD] token to be replaced by <s> and the output of Wav2Vec2ForCTC model to contain the “<s>” token.
Fix in training script: fix: Vocabulary build issue in run_speech_recognition.py · shivamm7/transformers@59635cb · GitHub 2