WhisperForAudioClassification can't be prepared for int8 training 
### System Info

System: Google Colab Pro

latest version of transformers installed through:
!pip install -q git+https://github.com/huggingface/transformers.git@main

### Who can help?

The code that I used is: 

```
from transformers import WhisperFeatureExtractor
from transformers import WhisperTokenizer
from transformers import WhisperProcessor
from transformers import WhisperForAudioClassification

# Select CUDA device index
import os

os.environ["CUDA_VISIBLE_DEVICES"] = "0"
model_name_or_path = "openai/whisper-small"

feature_extractor = WhisperFeatureExtractor.from_pretrained(model_name_or_path)
tokenizer = WhisperTokenizer.from_pretrained(model_name_or_path)
processor = WhisperProcessor.from_pretrained(model_name_or_path)

model = WhisperForAudioClassification.from_pretrained(model_name_or_path, load_in_8bit=True, device_map="auto" , num_labels=2, label2id=label2id, id2label=id2label)

from peft import prepare_model_for_int8_training
model = prepare_model_for_int8_training(model)
```

and the error I got is :
![image](https://user-images.githubusercontent.com/68870951/224712923-41d626a7-e326-4db4-aca4-9fda26ee0992.png)


@sanchit-gandhi


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce:

1. Open colab
2. Install latest version of transformers and peft 
```
!pip install -q git+https://github.com/huggingface/transformers.git@main
!pip install git+https://github.com/huggingface/peft.git@main
```
3. use the code
```
from transformers import WhisperFeatureExtractor
from transformers import WhisperTokenizer
from transformers import WhisperProcessor
from transformers import WhisperForAudioClassification

# Select CUDA device index
import os

os.environ["CUDA_VISIBLE_DEVICES"] = "0"
model_name_or_path = "openai/whisper-small"

feature_extractor = WhisperFeatureExtractor.from_pretrained(model_name_or_path)
tokenizer = WhisperTokenizer.from_pretrained(model_name_or_path)
processor = WhisperProcessor.from_pretrained(model_name_or_path)

model = WhisperForAudioClassification.from_pretrained(model_name_or_path, load_in_8bit=True, device_map="auto" , num_labels=2, label2id=label2id, id2label=id2label)

from peft import prepare_model_for_int8_training
model = prepare_model_for_int8_training(model)
``` 

### Expected behavior

Expected to get a model which is prepared for int 8 training 
cc @younesbelkada  @pacman100  <|||||>https://github.com/huggingface/transformers/pull/22133 should fix the issue