How to run distributed training on multiple machines?
## Environment info


- `transformers` version: 4.3.0
- Platform: PyTorch
- Python version: 3.7
- PyTorch version (GPU?): 1.7.1
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

##Who can help: 
Examples:
- maintained examples (not research project or legacy): @sgugger, @patil-suraj

## Information

Model I am using (Bert, XLNet ...): Roberta

## To reproduce
The script I'm working with is https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py
I know that we can run the distributed training on multiple GPUs in a single machine by
`python -m torch.distributed.launch --nproc_per_node=8 run_mlm.py --sharded_dpp`

But what if I can multiple machines with multiple GPUs, let's say I have two machines and each is with 8 GPUs, what is the expected command to run on these 16 GPUs?

I'm only aware that the PyTorch official documentation is using the RPC: https://pytorch.org/tutorials/intermediate/dist_pipeline_parallel_tutorial.html<|||||>This is more of a question for the PyTorch GitHub then ours since this is a question about to use `torch.distributed.launch`. That's why I'll close the issue. Still, I can share the command I run on my side:
```
python -m torch.distributed.launch --nproc_per_node 8 \
    --nnodes 2 \
    --node_rank rank_of_your_machine \
    --master_addr main_machine_ip \
    --master_port open_port_on_main_machine \
    run_mlm.py \
    --sharded_ddp \
    --all_other_args_to_script
```
where `rank_of_your_machine` should be 0 for the main machine and 1 for the other one, `main_machine_ip` the IP of the machine of rank 0 and `open_port_on_main_machine` the port to use to communicate between the two machines.<|||||>Thanks, this is really helpful