Update examples/documentation for torch.distributed.run
Most examples now read local rank from a CLI argument `--local_rank`. This ensures for it to work when using the torch launch utility `torch.distributed.launch`. However, since recent torch versions, `launch` is deprecated. The new suggested way to run distributed code from CLI is with `torch.distributed.run`. The difference from before is that instead of automatically passing CLI arguments, it sets the rank **as an environment variable**. The examples therefore need to be updated to read the ENV variable instead of the CLI argument.

See https://pytorch.org/docs/stable/elastic/run.html#launcher-api for more.
I missed that the training arguments already have this covered:

https://github.com/huggingface/transformers/blob/bcc3f7b6560c1ed427f051107c7755956a27a9f2/src/transformers/training_args.py#L681-L685

Documentation may need an update though (see below).<|||||>Hi @BramVanroy! Let us know if this isn't sufficiently documented so that we may patch the issue<|||||>@LysandreJik I think the code in the TrainingArgs is fine. Documentation may need an update as it refers to `launch` in a couple of places. A quick search gives me mentions of `torch.distributed.launch` in these documents that may need to be replaced by `torch.distributed.run`. You'd also need to check whether `--nproc_per_node` is still an accepted argument for that command because I am not sure. So it is not only a find-and-replace update!

- https://huggingface.co/transformers/performance.html
- https://huggingface.co/transformers/main_classes/deepspeed.html
- https://huggingface.co/transformers/debugging.html<|||||>Maybe of interest to @sgugger <|||||>I wouldn't update the documentation as it won't work for older versions of PyTorch and I would prefer to only put one command vs If your PyTorch version is this then xxx else yyy. Let's wait a bit more :-)<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>My previous comment still stands :-)<|||||>Definitely. I think it's a good idea to keep bumping this to open, so that others can also find this issue easily, and it can serve as a reminder that as soon as `launch` is obsolete, the docs need an update. It's a tiny thing and easy to forget, so an open issue may help to remember as time goes on.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.