KeyError: 'input_ids' on Whisper training with include_inputs_for_metrics
### System Info

- `transformers` version: 4.32.0.dev0
- Platform: Linux-5.15.109+-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.0 (gpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@sanchit-gandhi

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

This link https://colab.research.google.com/drive/1mMolQVClnnC_hi1J6DeCUOElJtatSXoz?usp=sharing points to a Colab notebook that reproduces the issue. It is a slightly modified version from the official post in https://huggingface.co/blog/fine-tune-whisper

Basically, whenever setting `include_inputs_for_metrics=True` for a training with whisper, the error message and stack trace below appear. 

As a clarification, I was just experimenting with `include_inputs_for_metrics`. I realize it doesn't make much sense in this context to receive the bare inputs in `compute_metrics`. It would be better to receive any other type of metadata from other dataset fields.

Nevertheless, it seems like this is a bug, given that these models receive their inputs as `input_features` and not `input_ids`.

```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
[<ipython-input-20-3435b262f1ae>](https://localhost:8080/#) in <cell line: 1>()
----> 1 trainer.train()

6 frames
[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   1524             self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size
   1525         )
-> 1526         return inner_training_loop(
   1527             args=args,
   1528             resume_from_checkpoint=resume_from_checkpoint,

[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in _inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
   1886                     self.control = self.callback_handler.on_step_end(args, self.state, self.control)
   1887 
-> 1888                     self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
   1889                 else:
   1890                     self.control = self.callback_handler.on_substep_end(args, self.state, self.control)

[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in _maybe_log_save_evaluate(self, tr_loss, model, trial, epoch, ignore_keys_for_eval)
   2211                     metrics.update(dataset_metrics)
   2212             else:
-> 2213                 metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
   2214             self._report_to_hp_search(trial, self.state.global_step, metrics)
   2215 

[/usr/local/lib/python3.10/dist-packages/transformers/trainer_seq2seq.py](https://localhost:8080/#) in evaluate(self, eval_dataset, ignore_keys, metric_key_prefix, **gen_kwargs)
    157         self._gen_kwargs = gen_kwargs
    158 
--> 159         return super().evaluate(eval_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
    160 
    161     def predict(

[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in evaluate(self, eval_dataset, ignore_keys, metric_key_prefix)
   2919 
   2920         eval_loop = self.prediction_loop if self.args.use_legacy_prediction_loop else self.evaluation_loop
-> 2921         output = eval_loop(
   2922             eval_dataloader,
   2923             description="Evaluation",

[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in evaluation_loop(self, dataloader, description, prediction_loss_only, ignore_keys, metric_key_prefix)
   3109             # Prediction step
   3110             loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
-> 3111             inputs_decode = self._prepare_input(inputs["input_ids"]) if args.include_inputs_for_metrics else None
   3112 
   3113             if is_torch_tpu_available():

[/usr/local/lib/python3.10/dist-packages/transformers/feature_extraction_utils.py](https://localhost:8080/#) in __getitem__(self, item)
     84         """
     85         if isinstance(item, str):
---> 86             return self.data[item]
     87         else:
     88             raise KeyError("Indexing with integers is not available when using Python based feature extractors")

KeyError: 'input_ids'
```

### Expected behavior

The dummy training should be able to complete its validation loop.
The `include_inputs_for_metrics=True` feature only supports text models, not other modalities for now.<|||||>Noted. Thanks for the quick response! I couldn't find anything that suggested so in the docs, so I hope this github issue comes in handy if anyone else tries this. 

I'll go ahead and close the issue.<|||||>You don't need to close this, we should probably support other modalities better in the Trainer ;-) I was mainly stating that `input_ids` is hard-coded in this code and we would need to update it.<|||||>Oh ok. I opened it again then. Thanks!<|||||>Should be fixed by the PR linked above if you want to try.<|||||>Hi. That's awesome! I just tried it in the colab notebook I shared and it worked nicely. Thanks!