WhisperForAudioClassification RuntimeError tensor size doesn't match
### System Info


- `transformers` version: 4.28.0
- Platform: Linux-5.4.190-107.353.amzn2.x86_64-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.13.4
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Multi-GPU on the same machine

### Who can help?

@sanchit-gandhi finetuning WhisperForAudioClassification model and getting this error. The dataset and script works fine when switching to the Wav2Vec2 model.

  File "/opt/conda/envs/voice/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 64, in _worker
    output = module(*input, **kwargs)
  File "/opt/conda/envs/voice/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jovyan/.local/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1739, in forward
    encoder_outputs = self.encoder(
  File "/opt/conda/envs/voice/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jovyan/.local/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 823, in forward
    hidden_states = inputs_embeds + embed_pos
RuntimeError: The size of tensor a (750) must match the size of tensor b (1500) at non-singleton dimension 1

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Followed this tutorial, and switched Wav2vec2 to WhisperForAudioClassification model.
Using a dataset with 100K+ wav files.
Training fails with the above error.

### Expected behavior

It would train with no errors.
Hey @xiao1ongbao! I'm hoping that your fine-tuning run was successful! Let me know if you encounter any issues - more than happy to help here 🤗<|||||>Hey! How did you solve that?<|||||>Do you have a reproducible code snippet to trigger the error @lnfin? I didn't encounter this in my experiments!