Whisper `get_prompt_ids` throws error when used with a 'FastTokenizer'
### System Info

- `transformers` version: 4.30.0.dev0
- Platform: macOS-13.0-arm64-arm-64bit
- Python version: 3.9.16
- Huggingface_hub version: 0.12.0
- Safetensors version: 0.2.8
- PyTorch version (GPU?): 1.13.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): 0.5.3 (cpu)
- Jax version: 0.3.6
- JaxLib version: 0.3.5
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@sanchit-gandhi @hollance 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```py
from transformers import WhisperTokenizerFast, WhisperTokenizer, GPT2Tokenizer, GPT2TokenizerFast


slow_tokenizer = WhisperTokenizer.from_pretrained('openai/whisper-tiny')
prompt_ids = slow_tokenizer.get_prompt_ids("Hello, world!", return_tensors="pt")
print('Whisper slow tokenizer succeeded')

try:
    fast_tokenizer = WhisperTokenizerFast.from_pretrained('openai/whisper-tiny')
    prompt_ids = fast_tokenizer.get_prompt_ids("Hello, world!", return_tensors="pt")
except Exception as e:
    print('Whisper fast tokenizer failed - ', e)
    

# Alternatively, this slow-fast param difference can be seen when tokenizing with a
# pipeline or any model that has a slow tokenizer `prepare_for_tokenization` method 
# that checks `add_prefix_space` (GPT2 is old but there are ~20 models this applies to)
tokenizer = GPT2Tokenizer.from_pretrained('gpt2', use_fast=False)
prompt_ids = tokenizer("Hello, world!", add_prefix_space=True)["input_ids"]
print('GPT2 slow tokenizer succeeded')

try:
    tokenizer = GPT2TokenizerFast.from_pretrained('gpt2')
    prompt_ids = tokenizer("Hello, world!", add_prefix_space=True)["input_ids"]
except Exception as e:
    print('Whisper fast tokenizer failed - ', e)
```

### Expected behavior

Are the slow and fast tokenizers supposed to have the same arg options for tokenizing text? They diverge with the `add_prefix_space` argument; while the slow tokenizer accepts and applies it with the [prepare_for_tokenization](https://github.com/huggingface/transformers/blob/3416bba7c70c358ac17efd3be31e9090135969ab/src/transformers/tokenization_utils.py#L502) method that same model's fast tokenizer does not and throws an error. Given that this arg difference appears to be present across all models where `add_prefix_space` can be provided to the slow tokenizer (at a glance appears to be ~20) I'd imagine the answer is no, the arg options aren't supposed to be 1:1.

The fix for the Whisper tokenizer `get_prompt_ids` method is straightforward as we can just do `" " + text` directly in the method instead of `add_prefix_space=True`, but I wanted to bring up the above in case that argument is actually supposed to compatible across both slow and fast tokenizers in which case we would also want to address that.
Related issue #17391 mentions that `add_prefix_space` can only be specified for fast tokenizers upon init, so it seems like just the manual `" " + text` replacement for this param would be the appropriate fix.<|||||>Hey! Thanks for reporting. Indeed I think you can easily fix this for a single model (in the fast tokenizer you could allow the argument to flow), but I do agreed that it is not really expected that the API between fast and slow would be different on that. 