Facing NCCL error on Multi-GPU training(on single machine) using run_glue.py script
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.2
- Platform: Linux-4.19.0-14-cloud-amd64-x86_64-with-debian-buster-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: 4xTesla T4 (GCP)
- Using distributed or parallel set-up in script?: torch.distributed.launch

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): DistilRoberta

The problem arises when using:
* [*] the official example scripts: (give details below)
* [  ] my own modified scripts: (give details below)


The tasks I am working on is:
* [  ] an official GLUE/SQUaD task: (give the name)
* [*] my own task or dataset: (give details below)

Regression task with a single output, using BertForSequenceClassification

## To reproduce

Steps to reproduce the behavior:

1.python -m torch.distributed.launch --nproc_per_node 4 /home/run_glue.py   --train_file /home/data/train.csv   --validation_file /home/data/dev.csv   --test_file /home/data/test.csv   --model_name_or_path distilroberta-base   --output_dir /home/model   --num_train_epochs 5   --per_device_train_batch_size 1   --per_device_eval_batch_size 16   --do_train   --do_eval   --fp16   --gradient_accumulation_steps 2   --do_predict   --logging_steps 100   --evaluation_strategy steps   --save_steps 100   --overwrite_output_dir

File "/opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 442, in init_process_group
732793de051f:1895:1925 [1] NCCL INFO transport/shm.cc:101 -> 2
732793de051f:1895:1925 [1] NCCL INFO transport.cc:30 -> 2
732793de051f:1895:1925 [1] NCCL INFO transport.cc:49 -> 2
732793de051f:1895:1925 [1] NCCL INFO init.cc:766 -> 2
732793de051f:1895:1925 [1] NCCL INFO init.cc:840 -> 2
732793de051f:1895:1925 [1] NCCL INFO group.cc:73 -> 2 [Async thread]
    barrier()
  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1947, in barrier
Traceback (most recent call last):
  File "/home/run_text_classification.py", line 480, in <module>
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1603729138878/work/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
    main()
  File "/home/run_text_classification.py", line 163, in main
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
  File "/opt/conda/lib/python3.7/site-packages/transformers/hf_argparser.py", line 180, in parse_args_into_dataclasses
    obj = dtype(**inputs)
  File "<string>", line 60, in __init__
  File "/opt/conda/lib/python3.7/site-packages/transformers/training_args.py", line 478, in __post_init__
    if is_torch_available() and self.device.type != "cuda" and self.fp16:
  File "/opt/conda/lib/python3.7/site-packages/transformers/file_utils.py", line 1346, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/transformers/training_args.py", line 583, in device
    return self._setup_devices

732793de051f:1897:1927 [3] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device
  File "/opt/conda/lib/python3.7/site-packages/transformers/file_utils.py", line 1336, in __get__
732793de051f:1897:1927 [3] NCCL INFO include/shm.h:41 -> 2

732793de051f:1897:1927 [3] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-b3d54cebe4167a34-0-2-3 (size 9637888)
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Expected model training to proceed smoothly using 4xGPU. When I run the said script with nproc_per_node=1(or even 2), it runs smoothly but setting it as 4 gives strange errors. 

After updating to 1.9.0 I face a different error:

RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:832, unhandled system error, NCCL version 2.7.8 ncclSystemError: System call (socket, malloc, munmap, etc) failed.
<!-- A clear and concise description of what you would expect to happen. -->

cc @sgugger <|||||>This seems like a problem in your environment install for NCCL: if your script can run on two GPUs there is nothing in the code to change to make it run on four GPUs so this is not a bug in the training script or transformers. I have never seen that particular NCCL error so I'm afraid I can't really help debugging it.<|||||>Thanks for the quick reply. 
Yeah, it’s strange that it works on 2 GPUs but not on 4. Will check again and let you know.<|||||>@sgugger just to clarify:
The system has 4 GPUs. It’s only the nproc_per_node argument I’m changing (from 1 to 2,4,etc.). 
Just want to ensure I’ve not misunderstood the cause of the error. Right?<|||||>Yes I understood that. The PyTorch launcher is going to spawn a different number of processes depending on the number your pass, which in turn will use the number of GPUs specified (and the others are idle).<|||||>Thanks. Just wanted to confirm that. Will try reinstalling the environment and update if I find the solution.
<|||||>Hi @sgugger,
Good news, the issue seems to have been an environment issue.
Thanks for the instant help<|||||>I still meet the same problem， could you please tell me how to solve it？