distributed training not starting
## 🐛 Bug

I'm using Huggingface's Transformers package, trying to start distributed training. But it is not starting as I run the distributed script. 

## To Reproduce

```
export CUDA_VISIBLE_DEVICES=0,1
python -m torch.distributed.launch --nproc_per_node=2 examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path facebook/bart-large \
    --do_predict \
    --output_dir $SAVE_MODEL_DIR/saved_models/$MODEL/$M_ID \
    --per_device_train_batch_size=1 \
    --per_device_eval_batch_size=8  \
    --learning_rate 3e-5 \
    --weight_decay 0.01 \
    --adam_beta2 0.98 \
    --num_train_epochs 10 \
    --overwrite_output_dir \
    --evaluation_strategy steps  --eval_steps 2000 --save_steps 2000 --warmup_steps 7000 --logging_steps 100 \
    --text_column intro \
    --summary_column summary \
    --train_file $DS_BASE_DIR/train.json \
    --validation_file $DS_BASE_DIR/val.json \
    --test_file $DS_BASE_DIR/test.json \
    --predict_with_generate

```

Steps to reproduce the behavior:

1. Running the provided script according to the instructions.
2. The code just stops working without any progress.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

I also tried monitoring the NCCL behaviour using `NCCL_DEBUG=INFO` flag. This is the output after running the script:

```
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
brunello:91348:91348 [0] NCCL INFO Bootstrap : Using [0]enp8s0:192.168.10.200<0> [1]br-494482ca770a:172.18.0.1<0>
brunello:91348:91348 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation

brunello:91348:91348 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
brunello:91348:91348 [0] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.10.200<0> [1]br-494482ca770a:172.18.0.1<0>
brunello:91348:91348 [0] NCCL INFO Using network Socket
NCCL version 2.7.8+cuda9.2
brunello:91349:91349 [1] NCCL INFO Bootstrap : Using [0]enp8s0:192.168.10.200<0> [1]br-494482ca770a:172.18.0.1<0>
brunello:91349:91349 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation

brunello:91349:91349 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
brunello:91349:91349 [1] NCCL INFO NET/Socket : Using [0]enp8s0:192.168.10.200<0> [1]br-494482ca770a:172.18.0.1<0>
brunello:91349:91349 [1] NCCL INFO Using network Socket
brunello:91348:91394 [0] NCCL INFO Channel 00/02 :    0   1
brunello:91349:91395 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
brunello:91348:91394 [0] NCCL INFO Channel 01/02 :    0   1
brunello:91349:91395 [1] NCCL INFO Trees [0] -1/-1/-1->1->0|0->1->-1/-1/-1 [1] -1/-1/-1->1->0|0->1->-1/-1/-1
brunello:91348:91394 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
brunello:91348:91394 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1|-1->0->1/-1/-1 [1] 1/-1/-1->0->-1|-1->0->1/-1/-1
brunello:91348:91394 [0] NCCL INFO Setting affinity for GPU 0 to ffffff
brunello:91349:91395 [1] NCCL INFO Channel 00 : 1[42000] -> 0[a000] via P2P/IPC
brunello:91348:91394 [0] NCCL INFO Channel 00 : 0[a000] -> 1[42000] via P2P/IPC
brunello:91348:91394 [0] NCCL INFO Channel 01 : 0[a000] -> 1[42000] via P2P/IPC
brunello:91349:91395 [1] NCCL INFO Channel 01 : 1[42000] -> 0[a000] via P2P/IPC
brunello:91348:91394 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
brunello:91349:91395 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
brunello:91348:91394 [0] NCCL INFO comm 0x7f1428001010 rank 0 nranks 2 cudaDev 0 busId a000 - Init COMPLETE
brunello:91349:91395 [1] NCCL INFO comm 0x7f8420001010 rank 1 nranks 2 cudaDev 1 busId 42000 - Init COMPLETE
brunello:91348:91348 [0] NCCL INFO Launch mode Parallel
```


## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

```
Collecting environment information...
PyTorch version: 1.7.1
CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.1 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: version 3.12.4
Libc version: glibc-2.7

Python version: 3.8.8
Python platform: Linux-4.15.0-43-generic-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: Yes
CUDA runtime version: 10.1
GPU models and configuration: 
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti

Nvidia driver version: 440.33.01
cuDNN version: /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn.so.7

Versions of relevant libraries:
[pip] numpy==1.16.5
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               9.2                           0  
[conda] mkl                       2021.2.0           h06a4308_296  
[conda] mkl-service               2.3.0            py38h27cfd23_1  
[conda] mkl_fft                   1.3.0            py38h42c9631_2  
[conda] mkl_random                1.2.1            py38ha9443f7_2  
[conda] mypy_extensions           0.4.3                    py38_0  
[conda] numpy                     1.20.1           py38h93e21f0_0  
[conda] numpy-base                1.20.1           py38h7d8b39e_0  
[conda] numpydoc                  1.1.0              pyhd3eb1b0_1  
[conda] pytorch                   1.7.1           py3.8_cuda9.2.148_cudnn7.6.3_0    pytorch
[conda] torchaudio                0.7.2                      py38    pytorch
[conda] torchvision               0.8.2                 py38_cu92    pytorch
```

 - PyTorch Version (e.g., 1.0): 1.7.1
 - OS (e.g., Linux): Linux
 - How you installed PyTorch (`conda`, `pip`, source): conda
 - Python version: 3.8.8
 - CUDA/cuDNN version: 10.2 
 - Any other relevant information: I tried Pytorch 1.7 with CUDA versions: 9.2, 10.1, and 10.2 but got the same sticking behaviour I mentioned.

cc blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj

I could solve the problem by switching to PyTorch version 1.6 with CUDA 10.2. I guess that might be because of the NCCL version incompatibility in my OS environment. So closing this issue as it's been solved. 