Add `trace_device` argument to `smp.DistributedModel` call in `Trainer()`
### Feature request

Allow `trace_device` to be passed to `smp.DistributedModel` so that `Trainer` jobs that use SageMaker Model Parallel can support models that exceed a single GPU's memory. 

Tagging @philschmid and @patil-suraj, because I think you both are involved in the SageMaker work?

### Motivation

The `Trainer` class provides native support for the SageMaker Model Parallel (smp) library; however, it does not support specifying the device where model tracing is conducted at the beginning of a smp training job. Because the default trace device is GPU, it is not possible to train a model that cannot fit in (a single) GPU memory, which, of course exactly when you would want to use model parallelism. 

This can be fixed by passing [trace_device](https://sagemaker.readthedocs.io/en/v2.20.0/api/training/smd_model_parallel_pytorch.html) to the `smp.DistributedModel` [call](https://github.com/huggingface/transformers/blob/d0acc9537829e7d067edbb791473bbceb2ecf056/src/transformers/trainer.py#L1236) in `Trainer()`. 

The value of `trace_device` could be specified in the [same way other smp parameters are specified](https://github.com/huggingface/transformers/blob/acb709d55150501698b5b500ca49683b913d4b3d/src/transformers/training_args.py#L904), which is via a json string of smp parameters, `mp_parameters`. 

As noted [elsewhere](https://github.com/huggingface/transformers/issues/14851#issuecomment-1013422175_), `trace_device` is not currently supported in the HF DLC, but it is apparently roadmapped. 

Accordingly, my current workaround is to use a SageMaker pytorch estimator and a custom Trainer class with the necessary overrides to ensure that `trace_device` is passed. 

It would be nice to be able to use the base trainer class rather than this workaround. 

### Your contribution

If this seems reasonable, I'd be happy to open a PR. 
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.