Multi-node training for casual language modeling example does not work
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.7.0.dev0
- Platform: Linux-4.19.0-14-amd64-x86_64-with-debian-10.8
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes 

### Who can help

@sgugger 

@patrickvonplaten, @LysandreJik 


## Information

Model I am using (Bert, XLNet ...): GPT-2

The problem arises when using:
* [x] my own modified scripts: 
```
nproc_per_node=4

  python -m torch.distributed.launch \
      --nproc_per_node=$nproc_per_node \
      --nnodes=2 \
      --node_rank=0 \
      --master_addr="192.168.1.1" \
      --master_port=1234  run_clm.py \
      --model_name_or_path gpt2 \
      --block_size 256 \
      --dataset_name wikitext \
      --dataset_config_name wikitext-2-raw-v1 \
      --do_train \
      --do_eval \
      --overwrite_output_dir \
      --num_train_epochs 1 \
      --output_dir /tmp/test-clm
```



The tasks I am working on is:
* [x] an official GLUE/SQUaD task: language modeling
* [x] my own task or dataset: wikitext

## To reproduce

Steps to reproduce the behavior:

1. Have two nodes with at least 4 GPUs each. 
2. In the first machine, run the above script.
3. In the second machine, run a script same as above except with the flag `--node_rank=1` instead of `--node_rank=0`.

I have waited for almost 15 mins. Nothing has happened. The training did not get started. 

## Expected behavior

The training gets started.  


Are you sure the port is open between the two machines? Not having any output is usually a symptom of that. I've tried on my side and I can run the script on multi-nodes.<|||||>@sgugger Thanks for the reply :)

> Are you sure the port is open between the two machines?

yes I made sure by giving different port numbers; none of them worked and I got this message after 15 mins:

`RuntimeError: connect() timed out.` 

<|||||>No they need to have the same port number, otherwise they can't connect to each other.<|||||>Thank you very much. 

> No they need to have the same port number

Here I meant I gave the same port number to both sides, but I tried multiple times with some numbers to make sure the port is open :) 

But no worries, the issue is solved. I tried with the actual IP address of one of the machines and that solved the issue. <|||||>Glad you solved your issue!