Wrong pre-trained Whisper's BOS token?
### System Info

- `transformers` version: 4.30.2
- Platform: macOS-13.4-arm64-arm-64bit
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sanchit

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import WhisperTokenizer
WhisperTokenizer.from_pretrained("openai/whisper-tiny").bos_token
>> '<|endoftext|>'
```

### Expected behavior

Dear Gandhi,

From the [documentation](https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperTokenizer) and from what I expect from the Whisper tokenizer, `processor.tokenizer.bos_token` should be equal to `"<|startoftranscript|>"` when using one of the official vanilla Whisper model. Currently, it is equal to `"<|endoftext|>"`. Is it an intended behavior? What do you think?

On a different note, there is another weird behavior when encoding/decoding:

```python
tokenizer.encode(["<|startoftranscript|>"])
>> [50258, 50363, 50258, 50257]
processor.tokenizer.decode([50258, 50363, 50258, 50257])
>> '<|startoftranscript|><|notimestamps|><|startoftranscript|><|endoftext|>'
```

while I was expecting the last line to return `'<|startoftranscript|>'` only.

Yours sincerely,
Tony
Hi!

The first issue seems to be a feature of the whisper model. It has `<|endoftext|>` as token text for `bos`, `eos`, `pad` and `unk`. I see there are no dedicated tokens for `unk` and `pad`, so I think this is a feature of the model, and not a bug. If you look at the [original code](https://github.com/openai/whisper/blob/main/whisper/tokenizer.py), you can see that there is no dedicated token for `eos`, `bos`, `pad` or `unk`. This seems to indicate that these tokens are simply not used by the model.

The second issue is due to `add_special_tokens` being set to `True` by default. So this is not unexpected behavior.

```python
tokenizer.encode(["<|startoftranscript|>"], add_special_tokens=False)
>>> [50258]
tokenizer.decode([50258])
>>> '<|startoftranscript|>'
```


<|||||>cc @ArthurZucker <|||||>Hey, not entirely sure which part of the documentation you are referring to, but this is expected. The `bos_token` is not used to start a transcript. More details [here](https://huggingface.co/openai/whisper-base) about the starting tokens, and why we don't use this `bos`.<|||||>> Hi!
> 
> The first issue seems to be a feature of the whisper model. It has `<|endoftext|>` as token text for `bos`, `eos`, `pad` and `unk`. I see there are no dedicated tokens for `unk` and `pad`, so I think this is a feature of the model, and not a bug. If you look at the [original code](https://github.com/openai/whisper/blob/main/whisper/tokenizer.py), you can see that there is no dedicated token for `eos`, `bos`, `pad` or `unk`. This seems to indicate that these tokens are simply not used by the model.
> 
> The second issue is due to `add_special_tokens` being set to `True` by default. So this is not unexpected behavior.
> 
> ```python
> tokenizer.encode(["<|startoftranscript|>"], add_special_tokens=False)
> >>> [50258]
> tokenizer.decode([50258])
> >>> '<|startoftranscript|>'
> ```

Thanks, makes sense!<|||||>> Hey, not entirely sure which part of the documentation you are referring to, but this is expected. The `bos_token` is not used to start a transcript. More details [here](https://huggingface.co/openai/whisper-base) about the starting tokens, and why we don't use this `bos`.

According to this [part](https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperTokenizer.bos_token), we have:

> bos_token (str, optional, defaults to "<|startoftranscript|>") — The beginning of sequence token.

Which in my opinion is a bit confusing.

But I do understand your point and how I should handle the `<|startoftranscript|>` now. Thanks for the help!<|||||>I'll update the documentation to make it less confusing. The token used to store the ` "<|startoftranscript|>"` token is `decoder_start_token_id`. The `bos_token` is  pretty much unused, which is why it was set to the same as `eos_token`. 