Patch for FlanT5-XXL 8bit support
# What does this PR do?

Fixes #20287 .

In #20287 , 3 patches were proposed here: https://github.com/huggingface/transformers/issues/20287#issuecomment-1342219429

* Patch 3 is already covered by https://github.com/huggingface/transformers/pull/20683
* I found patch 2 is actually unnecessary, because there's already a cast to float16 here: https://github.com/younesbelkada/transformers/blob/68a894a5875bfd958b8254afd3bbb23db9c2e813/src/transformers/models/t5/modeling_t5.py#L258-L260 which also applies in this case as we keep `self.wo` in `float32`.
* This PR contains the patch 1, adjusted so it only applies a cast if the `hidden_states` actually has a different `dtype` from the `wo` weights.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [n/a] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@younesbelkada @sgugger 

_The documentation is not available anymore as the PR was closed or merged._<|||||>> Thanks so much for the fix @larsmennen ! I would personally advocate to focus only on `T5`, and we can add these patches later on if we figure out that the same issue occur for all subsidiary models! Can you revert the changes for longt5/perceiver & switch (ideally also keep the copy mechanism, so maybe add the `# Copied from` statements but use another model as t5 as reference (for e.g. for perceiver `# Copied from transformers.src.models.longt5. ...`) Also don't forget to run the styling changes ;) (`make fixup`) Thanks again!

That makes sense! done