Whisper model predicts "thank  you" or "you" on silence
### System Info

wave2vec2 models predict text even there is no speech. The model predicts "thank you" and "you" on silence or empty speech.

Python 3.10
Ubuntu 20.04
transformers==4.30.2

https://github.com/gradio-app/gradio/issues/4663#issue-1772508542

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Follow these steps:

1. install gradio and transformers.
2. use automatic speech recognition pipeline and wave2vec2 base model.
3. use a microphone and don't speak. 

It will keep predicting words on an empty stream. 

Code to reproduce it.
```python
from transformers import pipeline
import gradio as gr

p = pipeline("automatic-speech-recognition", model="openai/whisper-base")

def transcribe(audio, state=""):
    text = p(audio)["text"]
    state += text + " "
    return state, state


# Set the starting state to an empty string
gr.Interface(
    fn=transcribe,
    inputs=[
        gr.Audio(source="microphone", type="filepath", streaming=True),
        "state"
    ],
    outputs=[
        "textbox",
        "state"
    ],
    live=True).launch(share=True)
```

### Expected behavior

It should not predict words on an empty stream.
Hmm, this code snippet involves `gradio` which is usually out of the scope of the `transformers` GitHub issue pages. But trying to have a reproducible example with only `transformers` involved seems tricky in this case.

@sgugger @amyeroberts @ArthurZucker Any general comment on how we proceed in such cases?

<|||||>Hey! 
Gradio can be interfering indeed, especially given the `streaming = True`. A few questions would help:
- does this happen on silence for  multiple audio? 
- was "thank you"  said before? Seems like the example
- Can you provide an example audio file when this happens? 
- Did you try without gradio (just the pipeline on the audio)? 
This would help us a lot if you can have these informations! <|||||>yes
no
audios (silence, noise, recording)[audios.zip](https://github.com/huggingface/transformers/files/11882755/audios.zip)
Yes.

```python
p = pipeline("automatic-speech-recognition", model="openai/whisper-base")

p("silence.wav")['text']
/home/irfan/.pyenv/versions/3.10.10/envs/WhisperDemo/lib/python3.10/site-packages/transformers/generation/utils.py:1353: UserWarning: Using `max_length`'s default (448) to control the generation length. This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation.
  warnings.warn(
' you'
```<|||||>Same issue, using gradio and non-gradio wav sound sources. I've also seen the behavior in [Unity Undertone](https://leastsquares.io/docs/unity/undertone) , a Whisper package for Unity 3D. So it may be in Whisper and not the ASR pipeline. Maybe a few more switches to control returned info might help.
[Whisper: Decode with condition_on_previous_text=False](https://github.com/huggingface/transformers/issues/21467#top)](https://github.com/huggingface/transformers/issues/21467)
[A possible solution to Whisper hallucination](https://github.com/openai/whisper/discussions/679)