I have finetuned a wav2vec2-xls-r-300m model on Hindi language using the mozilla-foundation/common_voice_7_0 dataset. But when I infer the model using the ASR pipeline special tokens are not skipped. And even in the Huggingface demo special characters are retained.
Model: shivam/xls-r-hindi (shivam/xls-r-hindi · Hugging Face 1)
Another example of same issue in different model: Harveenchadha/vakyansh-wav2vec2-hindi-him-4200 (Harveenchadha/vakyansh-wav2vec2-hindi-him-4200 · Hugging Face 1)
The tokenizer of wav2vec2-xls-r-300m is “Wav2Vec2CTCTokenizer” and in the asr pipeline if “CTC” is present in tokenizer class name then skip_special_tokens is set to False (transformers/automatic_speech_recognition.py at v4.15.0 · huggingface/transformers · GitHub 1), because of this special tokens are included in the output when using asr pipeline.

Screenshot from 2022-01-21 11-18-50751×90 13.4 KB


Screenshot from 2022-01-21 17-14-32660×516 34.8 KB