Distil* hanging on torch.distributed.barrier()
### System Info

- `transformers` version: 4.31.0.dev0
- Platform: Linux-5.14.21-150400.24.55-default-x86_64-with-glibc2.31
- Python version: 3.10.10
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 1.13.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help?

@VictorSanh @sgugger 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I'm trying to run [Distil*](https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation) project with a custom dataset. After the preprocessing steps, I enter the following command to start the training (single-node multi-GPU):

```
CUDA_VISIBLE_DEVICES=0,1,2 N_GPU_NODE=3 N_NODES=1 NODE_RANK=0 python -m torch.distributed.launch \
    --nproc_per_node=3 \
    train.py \
    	--force \
    	--dump_path serialization_dir/my_first_training \
    	--data_file ./data/binarized_text.roberta-base.pickle \
    	--student_type roberta \
    	--student_config ./training_configs/distilroberta-base.json \
    	--student_pretrained_weights ~/higo/distilbert/serialization_dir/tf_roberta_048131723.pth \
    	--teacher_type roberta \
    	--teacher_name roberta-base \
    	--mlm \
    	--temperature 2.0 \
    	--alpha_ce 5.0 \
    	--alpha_mlm 2.0 \
    	--alpha_clm 0.0 \
    	--alpha_mse 0.0 \
    	--alpha_cos 1.0 \
    	--token_counts ./data/token_counts.binarized_text.roberta-base.pickle \
    	--freeze_pos_embs \
    	--freeze_token_type_embds \
    	--n_epoch 4 \
    	--batch_size 8 \
    	--gradient_accumulation_steps 256 \
    	--learning_rate 2e-4 \
    	--n_gpu 3 \
    	--seed 42
```
When the script arrives [here](https://github.com/huggingface/transformers/blob/a074a5d34d6411fb00e83a2ed30acf23d8c976b5/examples/research_projects/distillation/distiller.py#L343), it get stuck, the train does not start and after a given span of time, I got a timeout error. I tried to set higher timeout values (up to 3 hours), with no result. At this given point, my `nvidia-smi` is shown like this:

```
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100 80GB PCIe           Off| 00000000:4F:00.0 Off |                    0 |
| N/A   34C    P0               71W / 300W|   2479MiB / 81920MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A100 80GB PCIe           Off| 00000000:52:00.0 Off |                    0 |
| N/A   34C    P0               69W / 300W|   1851MiB / 81920MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA A100 80GB PCIe           Off| 00000000:CE:00.0 Off |                    0 |
| N/A   36C    P0               68W / 300W|   1827MiB / 81920MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   3  NVIDIA A100 80GB PCIe           Off| 00000000:D1:00.0 Off |                    0 |
| N/A   42C    P0               66W / 300W|  63151MiB / 81920MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A     69824      C   .../home/u021274/higo/myenv/bin/python     2476MiB |
|    1   N/A  N/A     69825      C   .../home/u021274/higo/myenv/bin/python     1848MiB |
|    2   N/A  N/A     69826      C   .../home/u021274/higo/myenv/bin/python     1824MiB |
|    3   N/A  N/A     97463      C   python3                                   63148MiB |
+---------------------------------------------------------------------------------------+
```

(Process at GPU 3 is from another researcher)

To me, the little amount of allocated memory seems odd. I honestly don't have a clue of what can be happening. Checked some other threads, but nothing helped to make things clear.

### Expected behavior

Start of single-node, multi-GPU distributed training.
This is probably a setup error in the environment, as it means that one of the processes does not properly ping the others at the barrier line, and then it hangs forever.<|||||>Is there a way to solve this setup error?<|||||>i have same problem