Support Transformer Engine and FP8 training
### Feature request

NVIDIA has proposed the FP8 tensor core along with a [Transformer Engine](https://github.com/NVIDIA/TransformerEngine) library that implements the corresponding kernels and mix precision strategies (i.e. the delay scaling strategy). I wonder if we have plan on supporting transformer engine here? This could make better use of the newest hardware.

### Motivation

Make better use of the newest hardware, especially H100.
There is work ongoing to add support to it in [Accelerate](https://github.com/huggingface/accelerate/tree/fp8_integration) first. Once this is tested and merged, we will also port it to the `Trainer`. For now we are hit by a regression problem we are trying to fix with the team at Nvidia.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>The fp8 is now supported by Ada, and it is included in Accelerate. Will work continue on including fp8 in Trainer?
@sgugger<|||||>The Trainer will soon use Accelerate, so this will come for free.<|||||>Pls update here once the trainer has supported it, thanks!<|||||>Looking forward to updates!<|||||>When all is function supported in transformers.TrainingArguments?