Inference of finetuned wav2vec2-xls-r-300m model using the ASR pipeline does not remove special tokens.
## Environment info
- `transformers` version: 4.15.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU?): 1.10.0+cu111 (True)
- Tensorflow version (GPU?): 2.7.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
@patrickvonplaten , @anton-l ,  @Narsil

Models:
- Wav2Vec2-XLS-R

Library:
- Pipelines

Model hub:
- https://huggingface.co/shivam/xls-r-hindi

## Information

Model I am using (Bert, XLNet ...): XLSR-Wav2Vec2 (wav2vec2-xls-r-300m)

The problem arises when using:
* [ ] the official example scripts: ASR pipeline (https://github.com/huggingface/transformers/blob/v4.15.0/src/transformers/pipelines/automatic_speech_recognition.py)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: Automatic Speech Recognition (https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0)

## To reproduce

Steps to reproduce the behavior:

1. Finetune the wav2vec2-xls-r-300m model on Hindi language using the mozilla-foundation/common_voice_7_0 dataset using either (https://github.com/huggingface/transformers/blob/master/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py) or (https://huggingface.co/blog/fine-tune-xlsr-wav2vec2)
2. Infer the model using the ASR pipeline or Huggingface demo.
3. After inference special tokens like \<s\> and [UNK] are not skipped and present in the output.
4. ![Screenshot from 2022-01-21 17-14-32](https://user-images.githubusercontent.com/64258696/150523939-26d0541f-1a73-4172-ba5f-39c6206a583d.png)
5. The tokenizer of wav2vec2-xls-r-300m is “Wav2Vec2CTCTokenizer” and in the asr pipeline if “CTC” is present in tokenizer class name then skip_special_tokens is set to False (transformers/automatic_speech_recognition.py at v4.15.0 · huggingface/transformers · GitHub), because of this special tokens are included in the output when using asr pipeline.
6. ![Screenshot from 2022-01-21 11-18-50](https://user-images.githubusercontent.com/64258696/150523993-7b1e05cd-fa61-4d56-9067-9193c2d97ee5.png)

## Expected behavior

1. After inference the special tokens should be skipped in the output and the output should not contain special tokens. 

We need to not skip special tokens for CTC (wav2vec2 in particular) because of the [PAD] token.

HELLO can only be transcribed because the CTC tokens are H, E, L, PAD, L, L, L, O, O for instance.

It seems here that maybe `<s>` got confused with `<pad>` and hence is not properly skipped during decoding. Could that be it ? Also I am unsure to know how it would behave on the unicode scripts you are using (I hope if we remove `<s>` by properly using the `<pad>` token everything will work magically, but I can’t be sure.

That being said, making PAD be extra special in some regard, might be interesting.<|||||>Hey @shivamm7,

Could you post a code snippet that shows you error and that we could use to debug the model? :-) 

It could be as simple as:

```python
import torch
from datasets import load_dataset
from transformers import AutoModelForCTC, AutoProcessor
import torchaudio.functional as F


model_id = "shivam/xls-r-hindi"

sample = next(iter(load_dataset("common_voice", "hi", split="test", streaming=True)))

resampled_audio = F.resample(torch.tensor(sample["audio"]["array"]), 48_000, 16_000).numpy()

model = AutoModelForCTC.from_pretrained(model_id)
processor = AutoProcessor.from_pretrained(model_id)

input_values = processor(resampled_audio, return_tensors="pt").input_values

with torch.no_grad():
    logits = model(input_values).logits

prediction_ids = torch.argmax(logits, dim=-1)
transcription = processor.batch_decode(prediction_ids)

# TODO: What do you expect here?
```<|||||>I found the issue. The issue is with the vocabulary building procedure in run_speech_recognition_ctc.py (https://github.com/huggingface/transformers/blob/master/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py#L332).

While building vocabulary we replace the blank token " " by pipe symbol as delimiter ("|"). But if the dataset already contains the pipe symbol, which is common in Hindi language as Hindi language uses a similar character "।" as a full stop, in this case we will get two pipe symbols in the vocabulary. This will mess up the further process of creating the dictionary which is adding special tokens. Specifically, the bos token "\<s\>" and the pad token "[PAD]" will get the same index in vocabulary. This will cause the index with [PAD] token to be replaced by \<s\> and the output of Wav2Vec2ForCTC model to contain the "\<s\>" token.   

I have created a pull request #15285 which fixes this issue. 

Code snippet to recreate the issue:
```python
from datasets import Audio, Dataset, load_dataset, load_metric
from transformers import AutoFeatureExtractor, pipeline

dataset = load_dataset("mozilla-foundation/common_voice_7_0", "hi", split="test", use_auth_token=True)

# for testing: only process the first two examples as a test
dataset = dataset.select(range(10))

# load processor
feature_extractor = AutoFeatureExtractor.from_pretrained("shivam/xls-r-hindi")
sampling_rate = feature_extractor.sampling_rate

# resample audio
dataset = dataset.cast_column("audio", Audio(sampling_rate=sampling_rate))

# load eval pipeline
asr = pipeline("automatic-speech-recognition", model="shivam/xls-r-hindi")

# map function to decode audio
def map_to_pred(batch):
    prediction = asr(
        batch["audio"]["array"])

    batch["prediction"] = prediction["text"]
    batch["target"] = batch["sentence"]
    return batch

# run inference on all examples
result = dataset.map(map_to_pred, remove_columns=dataset.column_names)
print(result["prediction"])
```
This is the same code used in eval.py (https://github.com/huggingface/transformers/blob/master/examples/research_projects/robust-speech-event/eval.py)<|||||>Answered on the PR: https://github.com/huggingface/transformers/pull/15285 :-)<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Hey - this issue still exists in transformers `4.30.2`

```python
from transformers import pipeline
import torch
import requests

pipe = pipeline(
    "automatic-speech-recognition", 
    model="Harveenchadha/vakyansh-wav2vec2-bhojpuri-bhom-60", 
    device="cuda:0",
    torch_dtype=torch.float16,
)
audio = requests.get("https://storage.googleapis.com/dara-c1b52.appspot.com/daras_ai/media/54dc79c6-1c96-11ee-b5a1-02420a00015d/1min%201.wav").content
pipe(inputs=audio)
```

```
{'text': '<s>ा<s>ी<s>य<s>के<s> द<s> <s>आ<s>य<s> <s>ब<s>स<s> <s>ी<s> <s>स<s>े<s> <s>मि<s>ल<s>े<s> <s> <s>य<s>त<s>ना<s> <s>न<s> <s>आ<s>गा<s> <s>आ<s>ग<s> <s>ह<s> <s>द<s>ख<s>े<s> <s>े<s> <s>दि<s>दी<s> <s>क्य<s>ा<s>र<s>र<s>ह<s>ी<s> <s>ह<s> <s> <s>ह<s>म<s> <s>को<s> त<s>ो<s> <s>ज<s>ब<s> <s>कि<s>स<s>ा<s>न<s>ी<s> <s>े<s> <s>ह<s> क<s> ह<s>म<s> <s>क<s>र<s>र<s> <s> <s> <s>भ<s> <s>य<s>ा<s>थ<s>र<s>ा<s> <s>ध<s>ु<s>ने<s> आ<s>ह<s> <s>थ<s>र<s>ब<s>ब<s>ठ<s> <s>जा<s> <s>त<s>ो<s> <s>दी<s>दी<s> <s>ह<s>म<s> <s>स<s>ु<s>न<s>े<s> ह<s>ै<s> <s>कि<s> आ<s>प<s> <s>स<s>ि<s>र<s>ी<s> <s>भी<s> <s>धी<s> <s>से<s> भ<s>ी<s> <s>ख<s>ती<s> <s>क<s>र<s>त<s>े<s> <s>ह<s> <s>श<s>री<s> व<s>ि<s>धी<s>स<s>ो<s> <s>तो<s> <s>ह<s>म<s> <s>क<s>र<s> <s>र<s>ह<s>है<s> आ<s>प<s> <s>भी<s> <s>त<s>ो<s> <s>ख<s>े<s>ती<s> <s>क<s> र<s>ह ह<s>ैं त<s>ो<s> <s>अ<s>प<s>न<s> <s>ज<s>ा<s>न<s>ते<s> <s>ै<s> <s>न<s>ही<s> <s>न<s>ही<s> <s>ह<s> <s>तो<s> <s>ज<s>न<s> भी<s> <s>न<s>ही<s> <s>क<s>र<s>त<s>े<s> <s>है<s>ं<s>ख<s>त<s>ी<s>म<s>ने<s> <s>ह<s>म<s> <s>ल<s>ो<s>ग<s> <s>क<s>र<s>ते<s> ह<s>ै<s>ं<s> <s>त<s>ो<s>ब<s>ी<s>ज<s> उ<s>प<s>च<s>ा<s>र<s> <s>क<s>र<s>न<s>ा<s> <s>प<s>र<s>त<s>ा<s> है<s> <s>म<s>ी<s>ज<s>प<s>च<s>ा<s>र<s> <s>क<s>र<s>ना<s> <s>इ<s>स<s>के<s> <s>बा<s>रे<s> में न<s>हीं <s>प<s>ता<s> <s>है <s>प<s>को<s>न<s> <s>इ<s>स<s>क<s>े<s> <s>बा<s>रे<s> मे<s>ं <s>त<s> ह<s>ब<s>त<s>ा<s> <s>क<s>र<s>ना<s> <s>च<s>ा<s>हि<s> <s>प<s>ह<s>त<s>ना<s> <s>दि<s>न<s> स<s>े<s> <s>ख<s>े<s>ती<s> <s>क<s>र<s> र<s>ह<s>है<s> आ<s>प<s>को<s> <s>प<s>त<s>े<s> <s>न<s>ही<s> <s>ह<s>ै<s> <s>ज<s>ो<s> <s>बी<s>ज<s> उ<s>च<s>ा<s>र<s> <s>क<s>र<s> <s>के<s> <s>ख<s>े<s>त<s>ी<s> <s>क<s>र<s>ते<s> ह<s>ै<s>ं<s> <s>त<s>ो<s> <s>क<s>स<s>फ<s>स<s>ल<s> हो<s>दी<s> <s>र<s>ा<s>ई<s>ी<s> <s>के<s> <s>ब<s>ा<s>र<s>े<s> <s>मे<s>ं<s> <s>र<s> <s>के<s> <s>दे<s>ख<s>ा<s>द<s>े<s> <s>त<s>े<s> <s>तो<s> अ<s>च्<s>छ<s>ा<s> <s>र<s>ह<s>ा<s>'}
```<|||||>Here's my grand hack to make this work - 

```python
import numpy as np

def postprocess(model_outputs):
    final_items = []
    key = "tokens"
    for outputs in model_outputs:
        items = outputs[key].numpy()
        final_items.append(items)
    items = np.concatenate(final_items, axis=1)
    items = items.squeeze(0)
    return { "text": pipe2.tokenizer.decode(items, skip_special_tokens=False) }

pipe.postprocess = postprocess
```