Seq2Seq Trainer for QA: No useful metric returned for evaluation
### System Info

Latest Transformers version from main.

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi,

unfortunately, the `run_seq2seq_qa.py` in PyTorch Example folder does not output a useful evaluation metric:

```bash
python run_seq2seq_qa.py \
  --model_name_or_path google/mt5-small \
  --dataset_name mlqa \
  --dataset_config mlqa-translate-train.de \
  --context_column context \
  --question_column question \
  --answer_column answers \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 1 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --save_steps 5000 \
  --max_steps 50 \
  --output_dir ./mt5-small
```

I'm running this command and the final output looks like:

```
***** eval metrics *****
  epoch                   =       0.01
  eval_loss               =    15.7694
  eval_runtime            = 0:00:48.60
  eval_samples            =      10584
  eval_samples_per_second =    217.741
  eval_steps_per_second   =     27.218
```

So I'm missing EM and F1-Score: why is it no longer there? This also happens when fine-tuning SquAD 2.0.

### Expected behavior

Useful evaluation metric: EM and F1-Score should be returned.
You need to pass along `--predict_with_generate` to use generate in the evaluation, and then get the metrics.<|||||>Thanks @sgugger ! I used that option, but after evaluation it outputs:

```bash
Traceback (most recent call last):
  File "run_seq2seq_qa.py", line 724, in <module>
    main()
  File "run_seq2seq_qa.py", line 683, in main
    metrics = trainer.evaluate(max_length=max_length, num_beams=num_beams, metric_key_prefix="eval")
  File "/home/ubuntu/transformers/examples/pytorch/question-answering/trainer_seq2seq_qa.py", line 92, in evaluate
    eval_preds = self.post_process_function(eval_examples, eval_dataset, output)
  File "run_seq2seq_qa.py", line 617, in post_processing_function
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
  File "/home/ubuntu/transformers/src/transformers/tokenization_utils_base.py", line 3445, in batch_decode
    return [
  File "/home/ubuntu/transformers/src/transformers/tokenization_utils_base.py", line 3446, in <listcomp>
    self.decode(
  File "/home/ubuntu/transformers/src/transformers/tokenization_utils_base.py", line 3485, in decode
    return self._decode(
  File "/home/ubuntu/transformers/src/transformers/tokenization_utils_fast.py", line 549, in _decode
    text = self._tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)
OverflowError: out of range integral type conversion attempted
```<|||||>Ah, I've just found this issue for that problem: https://github.com/huggingface/transformers/issues/22634

So I'll close here!