Whisper model adds "!" char in the beginning of each predicted audio transcription
### System Info

Google Colab instance with Tesla T4

- `transformers` version: 4.26.0.dev0
- Platform: Linux-5.10.147+-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.1+cu116 (True)
- Tensorflow version (GPU?): 2.9.2 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@sanchit-gandhi @gante @Narsil 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Here is the link to [Google Colab notebook](https://colab.research.google.com/drive/1VrcO9BW4OSEm8jmQjvzZ8vikMLsX5n3n?usp=sharing)

```python
!pip install git+https://github.com/huggingface/transformers

from transformers import pipeline

pipe = pipeline(
    task="automatic-speech-recognition",
    model='ales/whisper-small-belarusian',
    chunk_length_s=8, stride_length_s=1, device=0,
)

pipe.model.config.forced_decoder_ids = pipe.tokenizer.get_decoder_prompt_ids(
    language='be', task='transcribe'
)

# run with transformers installed from latest commit: 00ba7cadd812437708b380ab078a3cfe8cfaff31 at the moment.
# all transcription have extra "!" in the beginning!
res = pipe('audio_sample.ogg')
res2 = pipe('audio_sample2.ogg')
print(res)
print(res2)
```

output:
```
{'text': '!Хацеў бы спаткацца з вамі на вуліцу ціхаю зорнаю ночы і сказаць, ці бачыце гэтыя зоркі, ясныя зоркі, іграб лес.!'}
{'text': '!Прывітанне, як вашыя справы.'}
```

### Expected behavior

The problem is when using `ales/whisper-small-belarusian` Whisper model that I've fine-tuned from `openai/whisper-small` each transcription that model now produces starts with an exclamation mark ("!"). This look like an error in model decoding.

* The problem occured when I upgraded my environment.  I use `git+https://github.com/huggingface/transformers` as a version specifier to install `transformers` from source. The reason to install `transformers` from source is that current latest release `v4.25.1` does not have needed functionality (e.g. `WhisperTokenizer` class does not have `get_decoder_prompt_ids` method)
* Current latest commit in the `transformers` repository is `00ba7cadd812437708b380ab078a3cfe8cfaff31`
* When I deleted my Colab runtime and created a new one, now installing `transformers` from an older commit `a081f292ca8479eaf66d7396186021268f128829`, transcriptions returned to normal - no exclamation mark in the beginning.
* I guess this error was introduced by some latest Pull requests after `a081f292ca8479eaf66d7396186021268f128829` commit

Compare 2 transcriptions (examples could be found above and in the [Google Colab notebook](https://colab.research.google.com/drive/1VrcO9BW4OSEm8jmQjvzZ8vikMLsX5n3n?usp=sharing))
* Transcription with `transformers` installed from the latest commit (`00ba7cadd812437708b380ab078a3cfe8cfaff31`):
    ```
    {'text': '!Прывітанне, як вашыя справы.'}
    ```
* Transcription if we use older commit (`a081f292ca8479eaf66d7396186021268f128829`):
    ```
    {'text': 'Прывітанне, як вашыя справы.'}
    ```

And this happens to any audiofile I pass to the pipeline.

Thanks!
@ArthurZucker 

Could it be the timestamps PR ? We did change the `force_bos_token_ids` but only when we want timestamps, right ?<|||||>I think it related to that indeed, I'll have a look!<|||||>Okay so the `WhisperTimestampProcessor` is always added to the list of logit processors. This is the cause of the error 😉 
See this script were I added 
```python 
            return_timestamps = generate_kwargs.pop("return_timestamps", False)
            tokens = self.model.generate(
                input_features=model_inputs.pop("input_features"),
                logits_processor=[WhisperTimeStampLogitsProcessor()] if return_timestamps else None,
                **generate_kwargs,
            )
```
in the `_forward` call of pipeline. 

```python 
from transformers import pipeline
from datasets import load_dataset

libri = load_dataset("librispeech_asr", f"clean", split="test", cache_dir="/home/arthur_huggingface_co/.cache/huggingface/datasets")

pipe = pipeline(
    task="automatic-speech-recognition",
    model='openai/whisper-tiny',
    chunk_length_s=8, stride_length_s=1, device=0,
)

pipe.model.config.forced_decoder_ids = pipe.tokenizer.get_decoder_prompt_ids(
    language='fr', task='transcribe'
)

res = pipe(libri[0]["audio"]["array"], return_timestamps=False)
```<|||||>Two possible fixes : 
- both the forward and the initialisation should be consistent. So the `return_timestamp` arg should be added to the `self.args`. 
- Just add this to the generation config or the parameters of the _forward call as I did. 
WDYT @Narsil <|||||>What about just keeping `return_timestamps` and just sending it to both `preprocess` and `_forward` ?

It seems odd to push it into `generate_kwargs` if the generation doesn't care about it (directly I mean)