Fix integration with Accelerate and failing test
# What does this PR do?

This PR brings back the logic when gathering and calculating metrics, which was borked with https://github.com/huggingface/transformers/pull/24028. Proof is the fact that tests now pass that were failing related to the Trainer

Fixes # (issue)

Solves https://github.com/huggingface/transformers/issues/24391

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

As Sylvain is on vacation, cc @amyeroberts 

_The documentation is not available anymore as the PR was closed or merged._<|||||>@muellerzr I'm not sure if it's this or the previous PR that is causing the issue, but it still seems to be hanging on moving the loss tensor to the cpu.<|||||>@winglian can you provide a reproducer?<|||||>I'll have to distill something down later, but I can confirm the issue happens on multi-gpu, but when using single gpu, it seems to work properly.<|||||>A repr will definitely be needed here, because so far at least all the official example scripts don't hang for me. (Though there was something with gradient accumulation fixed with https://github.com/huggingface/transformers/pull/24756). Ping me once you have that and I can take a look. (Though sooner is better so I can try and have it for the next release 😄 )