`prediction_loss_only` = False returns `float.detatch()` error witrh HF trainer
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.10.2
- Platform:
- Python version: 3.7
- PyTorch version (GPU?): 1.9.0
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, mBERT...):

The problem arises when using:
* [x ] my own modified scripts: override `predict()` and `evaluate()` class

The tasks I am working on is:
* [x ] my own task or dataset: XTREME multilingual tasks

## To reproduce

I was using transformers v2.11 and it worked fine. I upgraded to the current version of the `transformers` for using `load_best_model_at_end`. It appears when I set flag `prediction_loss_only = True` there is no error. I logged the output and it is returning all the tensors. I don't why there is a `float.detach()` error.



```
File "/student/mda219/garage/miniconda3/envs/py3gpu/lib/python3.7/site-packages/transformers/trainer.py", line 1340, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/student/mda219/garage/miniconda3/envs/py3gpu/lib/python3.7/site-packages/transformers/trainer.py", line 1445, in _maybe_log_save_evaluate
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/student/mda219/garage/miniconda3/envs/py3gpu/lib/python3.7/site-packages/transformers/trainer.py", line 1445, in _maybe_log_save_evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/data/data_store/rabiul/codes/mumt/camtl/src/mumt_trainer.py", line 295, in evaluate
    eval_dataset=eval_dataset,
  File "/student/mda219/garage/miniconda3/envs/py3gpu/lib/python3.7/site-packages/transformers/trainer.py", line 2058, in evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/data/data_store/rabiul/codes/mumt/camtl/src/mumt_trainer.py", line 295, in evaluate
    eval_dataset=eval_dataset,
  File "/student/mda219/garage/miniconda3/envs/py3gpu/lib/python3.7/site-packages/transformers/trainer.py", line 2058, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "/student/mda219/garage/miniconda3/envs/py3gpu/lib/python3.7/site-packages/transformers/trainer.py", line 2223, in evaluation_loop
    metric_key_prefix=metric_key_prefix,
  File "/student/mda219/garage/miniconda3/envs/py3gpu/lib/python3.7/site-packages/transformers/trainer.py", line 2223, in evaluation_loop
    loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
  File "/student/mda219/garage/miniconda3/envs/py3gpu/lib/python3.7/site-packages/transformers/trainer.py", line 2451, in prediction_step
    loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
  File "/student/mda219/garage/miniconda3/envs/py3gpu/lib/python3.7/site-packages/transformers/trainer.py", line 2451, in prediction_step
    logits = nested_detach(logits)
  File "/student/mda219/garage/miniconda3/envs/py3gpu/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 145, in nested_detach
    logits = nested_detach(logits)
  File "/student/mda219/garage/miniconda3/envs/py3gpu/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 145, in nested_detach
    return type(tensors)(nested_detach(t) for t in tensors)
  File "/student/mda219/garage/miniconda3/envs/py3gpu/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 145, in <genexpr>
    return type(tensors)(nested_detach(t) for t in tensors)
  File "/student/mda219/garage/miniconda3/envs/py3gpu/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 145, in <genexpr>
    return type(tensors)(nested_detach(t) for t in tensors)
  File "/student/mda219/garage/miniconda3/envs/py3gpu/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 146, in nested_detach
    return type(tensors)(nested_detach(t) for t in tensors)
  File "/student/mda219/garage/miniconda3/envs/py3gpu/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 146, in nested_detach
    return tensors.detach()
AttributeError: 'float' object has no attribute 'detach'
    return tensors.detach()
AttributeError: 'float' object has no attribute 'detach'
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->


<!-- A clear and concise description of what you would expect to happen. -->

Hi,

Could you provide a code snippet that can reproduce this error? <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.