Trainer Eval loop fails to handle COCO formatted data 
### System Info

- `transformers` version: 4.23.1
- Platform: Linux-4.14.294-220.533.amzn2.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO

### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Download and format (into Dataset object) a COCO vision dataset with the following features:
```
     features = Features(
    {
        "pixel_mask": Sequence(Sequence(Sequence(Value(dtype="float32")))),
        'labels': [{
            'boxes': Sequence(Sequence(Value(dtype="float32"))),
            'class_labels': Sequence(ClassLabel(names=label_list)),
            'image_id': Sequence(Value(dtype="int64")),
            'area': Sequence(Value(dtype="float32")),
            'iscrowd': Sequence(Value(dtype="int64")),
            'orig_size': Sequence(Value(dtype="int64")),
            'size': Sequence(Value(dtype="int64"))
        }],
        'carrier': Value(dtype='string'),
        'pixel_values': Array3D(dtype="float32", shape=(3, 1049, 800))
    }
)
```
3. Load the DETR model
```
def model_init():
    return DetrForObjectDetection.from_pretrained('facebook/detr-resnet-101-dc5',
                                                         id2label=id2label,
                                                         label2id=label2id,
                                                 ignore_mismatched_sizes=True)
```
4. Create custom collator function
```
def collate_fn(batch):
    pixel_values = torch.cat([item["pixel_values"].unsqueeze(dim=0) for item in batch])
    encoding = feature_extractor.pad_and_create_pixel_mask(
        pixel_values, return_tensors="pt"
    )
    labels = [item["labels"][0] for item in batch]
    
    batch = {}
    batch['pixel_values'] = pixel_values
    batch["pixel_mask"] = encoding["pixel_mask"]
    batch["labels"] = labels
    return batch
```
5. Instantiate Trainer
```
training_args = transformers.TrainingArguments(
    output_dir=output_dir,
    logging_dir=logging_dir,
    max_steps=1000,
    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    learning_rate=1e-5,
    evaluation_strategy="steps",
    eval_steps=100,
    save_strategy='steps',
    save_steps=100,
    report_to="tensorboard",
    logging_strategy='steps',
    logging_steps=50,
    seed=42
)

trainer = Trainer(
    model_init=model_init,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    data_collator=collate_fn,
    compute_metrics=compute_metrics
    # callbacks=[tensorboard_callback]
)
```
6. Call trainer.prediction_step
`loss, logits, labels = trainer.prediction_step(model, example, False, ignore_keys=True)`

7. Receive error message:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In [86], line 1
----> 1 loss, logits, labels = trainer.prediction_step(model, example, False, ignore_keys=True)

File /opt/conda/lib/python3.8/site-packages/transformers/trainer.py:3166, in Trainer.prediction_step(self, model, inputs, prediction_loss_only, ignore_keys)
   3164 # labels may be popped when computing the loss (label smoothing for instance) so we grab them first.
   3165 if has_labels:
-> 3166     labels = nested_detach(tuple(inputs.get(name) for name in self.label_names))
   3167     if len(labels) == 1:
   3168         labels = labels[0]

File /opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py:158, in nested_detach(tensors)
    156 "Detach `tensors` (even if it's a nested list/tuple of tensors)."
    157 if isinstance(tensors, (list, tuple)):
--> 158     return type(tensors)(nested_detach(t) for t in tensors)
    159 return tensors.detach()

File /opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py:158, in <genexpr>(.0)
    156 "Detach `tensors` (even if it's a nested list/tuple of tensors)."
    157 if isinstance(tensors, (list, tuple)):
--> 158     return type(tensors)(nested_detach(t) for t in tensors)
    159 return tensors.detach()

File /opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py:159, in nested_detach(tensors)
    157 if isinstance(tensors, (list, tuple)):
    158     return type(tensors)(nested_detach(t) for t in tensors)
--> 159 return tensors.detach()

AttributeError: 'dict' object has no attribute 'detach'

### Expected behavior

The function should return the loss, logits, labels as expected. 
This has been fixed by #19455, you should upgrade to the latest version of Transformers to have the fix.