[WHISPER] Unreliable timestamp with whisper for videos under 30 seconds
### System Info

Hey, I noticed that there's an unreliable timestamp thing happening which whisper through transformers that doesn't show up in original whisper. 

In this example: 
https://targum.video/v/47160791e0e305ff7f22e84203f1b196

The "people on streches.." was said at the end of the video, but the timestamp was placed around 8 seconds in. 

Here's the same video translated with large-2 whisper from OpenAI

https://targum.video/v/8c5e21ff6da8947c02cdb40097eadf50

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Can send the exact link where this is happens over DM, but basically the video from this tweet: 

https://twitter.com/wxkaitlin/status/1625326828264071168

Is getting the wrong subtitles (some are missing) + a wrong timestamp

### Expected behavior

Expected the transformers whisper to behave exactly like (but faster) the openAI whisper
Can you provide a reproduction script to make sure we are running with the same parameters? 😉 
Also this might ring some bels to @Narsil. 
I know we interacted before, but just want to make sure which transformers version you are using and which calls. Then I'll be able to dig ! Thanks for the issue 😉 
<|||||>@altryne @ArthurZucker .

While deep diving into whisper, I've notived `openai/whisper` uses timestamp ALL the time, while `transformers` doesn't (you have to ask for timestamps for us to use them).

I have seen BIG discrepancies on some examples, I am guessing because training was somehow biased with timestamps for whisper. 

Could that be it ?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.