 Provides an option to select the parallel mode of the Trainer.
# 🚀 Feature request

Provides an option to select the parallel mode of the Trainer.

## Motivation

For multiple GPUs, Trainer uses `nn.DataParallel` for parallel computing by default, however, this approach results in a large memory occupation for the first GPU. Please provide an API to switch to `nn.parallel.DistributedDataParallel`. Also, for the `Trainer.predict()` function is there an option to turn off parallel computing?

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

@sgugger 
This is already implemented, it just depends on how you launch your training script. To use distributed data parallel, you have to launch it with `torch.distributed.launch`.<|||||>> This is already implemented, it just depends on how you launch your training script. To use distributed data parallel, you have to launch it with `torch.distributed.launch`.

Hi~
How to do for jupyter?
Also what should I do if I use DataParallel for train and only want to use one GPU for predict?
As the `DataParallel` requires droplast, which is not allowed in predict phase.
Thanks
<|||||>You can't do this directly in jupyter, you have to launch a script using the pytorch utilities (it's not a Trainer limitation, it's a PyTorch one). You can completely predict in parallel with the `Trainer`, it will complete the last batch to make it the same size as the others and then truncate the predictions.<|||||>> You can't do this directly in jupyter, you have to launch a script using the pytorch utilities (it's not a Trainer limitation, it's a PyTorch one). You can completely predict in parallel with the `Trainer`, it will complete the last batch to make it the same size as the others and then truncate the predictions.

But we can't truncate predictions because it's a contest or a client's demand and we need all the test results we can get.

As a demo, #11833
https://github.com/huggingface/transformers/issues/11833<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.