[Whisper] Potential inconsistencies across whisper tokenizers (`pad_token_id`)
### System Info

 `transformers` version: 4.27.2
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.3
- Huggingface_hub version: 0.12.0
- PyTorch version (GPU?): 1.13.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@sanchit-gandhi

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

There seems to be a mismatch between english and multilingual versions of the whisper models, which are present on the hub:

| Size     | Parameters | English-only                                         | Multilingual                                        |
|----------|------------|------------------------------------------------------|-----------------------------------------------------|
| tiny     | 39 M       | [✓](https://huggingface.co/openai/whisper-tiny.en)   | [✓](https://huggingface.co/openai/whisper-tiny)     |
| base     | 74 M       | [✓](https://huggingface.co/openai/whisper-base.en)   | [✓](https://huggingface.co/openai/whisper-base)     |
| small    | 244 M      | [✓](https://huggingface.co/openai/whisper-small.en)  | [✓](https://huggingface.co/openai/whisper-small)    |
| medium   | 769 M      | [✓](https://huggingface.co/openai/whisper-medium.en) | [✓](https://huggingface.co/openai/whisper-medium)   |
| large    | 1550 M     | x                                                    | [✓](https://huggingface.co/openai/whisper-large)    |
| large-v2 | 1550 M     | x                                                    | [✓](https://huggingface.co/openai/whisper-large-v2) |

---

For example, the tiny models:
- the [english version](https://huggingface.co/openai/whisper-tiny.en/blob/main/generation_config.json) uses a pad token of 50256
- the [multilingual version](https://huggingface.co/openai/whisper-tiny/blob/main/generation_config.json) uses a pad token of 50257

I would assume that both models would use similar vocabularies? I could be mistaken though. The multilingual model might of course include additional tokens for languages, but the pad_token (and other special tokens) would probably have the same IDs?

Moreover, the pad_token_ids were recently updated for multilingual versions (see [commit](https://huggingface.co/openai/whisper-tiny/commit/a8d76517e6d65d92771752dbbf5e9c0a1a5b3a0d)).

### Expected behavior

Tokenizers should use the same `pad_token_id`
Actually... looking at the vocabularies for each tokenizer:
- [multilingual](https://huggingface.co/openai/whisper-tiny/raw/main/vocab.json)'s token with id 50257 is `<|endoftext|>`, vs `""`, which has id 50256
- [english](https://huggingface.co/openai/whisper-tiny.en/raw/main/vocab.json)'s token with id 50256 is `<|endoftext|>` (and does not seem to have a `""` token)

---

I assume this is intended then. Will close the issue!<|||||>It is indeed intended! The English-only and multilingual tokenizers have different vocabulary items, and hence different index: vocab mappings, so the padding token is in different indices for each.