Data collator using in Parallel training & Disable to use DistributedDataParallel
### System Info

```shell
- `transformers` version: 4.12.0.dev0
- Platform: Linux-4.15.0-45-generic-x86_64-with-debian-stretch-sid
- Python version: 3.7.10
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Both are tried and failed
```


### Who can help?

Library/Trainer: @sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

code sample that reproduces the problem(with data): https://drive.google.com/file/d/1jFLV9Ir0Um3C6MTPjCgZMUKJtmasrQPT/view?usp=sharing

terminal input:
For DDP:
CUDA_VISIBLE_DEVICES=5,6 python -m torch.distributed.launch base-Trainer.py --mode "Data" --train_data_path utils/dev-small.json --output_dir outputs/Data_only_load_data/ --do_train --per_device_train_batch_size 4 --save_steps 100000
For DataParallel:
CUDA_VISIBLE_DEVICES=5,6 python base-Trainer.py --mode "Data" --train_data_path utils/dev-small.json --output_dir outputs/Data_only_load_data/ --do_train --per_device_train_batch_size 4 --save_steps 100000

some explaination to the task and files:
1. The task is MLM, with specified mask tokens, not randomly choosed.
2. tokenizer is basically a roberta-base-tokenizer, adding a special token [pron]
3. Dataset returns one sample of data each time, with type str or (str, str)
4. tokenization and label-creating are realized in collater()

error messages:
when using DataParallel:
/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/torch/nn/parallel/_functions.py:61: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.

when using DDP:
1. set 2 gpu devices but only one gpu is used.
5. terminal outputs as follows:
Traceback (most recent call last):
  File "base-Trainer.py", line 67, in <module>
    main()
  File "base-Trainer.py", line 62, in main
    trainer.train()
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/transformers/trainer.py", line 1383, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/transformers/trainer.py", line 1475, in _maybe_log_save_evaluate
    tr_loss_scalar = self._nested_gather(tr_loss).mean().item()
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/transformers/trainer.py", line 2385, in _nested_gather
    tensors = distributed_concat(tensors)
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 168, in distributed_concat
    dist.all_gather(output_tensors, tensor)
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1185, in all_gather
    work = _default_pg.allgather([tensor_list], [tensor])
RuntimeError: All tensor operands to scatter/gather must have the same size
  0%|▎                                                                                                                    | 500/183354 [02:03<12:32:13,  4.05it/s]
Traceback (most recent call last):
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/torch/distributed/launch.py", line 261, in <module>
    main()
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/torch/distributed/launch.py", line 257, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/home/caoyq/.conda/envs/torch_cp37/bin/python', '-u', 'base-Trainer.py', '--local_rank=0', '--mode', 'Data', '--train_data_path', 'utils/dev-small.json', '--output_dir', 'outputs/Data_only_load_data/', '--do_train', '--per_device_train_batch_size', '4', '--save_steps', '100000']' returned non-zero exit status 1.

### Expected behavior

```shell
1. I want to use DDP, not just DP
2. From the error messages given above, my data collator may have problems. It would be so nice of you if you could tell me what's wrong and how can I fixed it. If there exists a sample I can follow, please let me know.

best wish
```

Setting nproc_per_node=2 enables the multi-gpu training by ddp for the first 500 steps. But right after the 500th step, the same problem occurs as follows:
Traceback (most recent call last):
  File "base-Trainer.py", line 67, in <module>
    main()
  File "base-Trainer.py", line 62, in main
    trainer.train()
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/transformers/trainer.py", line 1383, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/transformers/trainer.py", line 1475, in _maybe_log_save_evaluate
    tr_loss_scalar = self._nested_gather(tr_loss).mean().item()
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/transformers/trainer.py", line 2385, in _nested_gather
    tensors = distributed_concat(tensors)
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 168, in distributed_concat
    dist.all_gather(output_tensors, tensor)
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1185, in all_gather
    work = _default_pg.allgather([tensor_list], [tensor])
RuntimeError: All tensor operands to scatter/gather must have the same size
Traceback (most recent call last):
  File "base-Trainer.py", line 67, in <module>
    main()
  File "base-Trainer.py", line 62, in main
    trainer.train()
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/transformers/trainer.py", line 1383, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/transformers/trainer.py", line 1475, in _maybe_log_save_evaluate
    tr_loss_scalar = self._nested_gather(tr_loss).mean().item()
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/transformers/trainer.py", line 2385, in _nested_gather
    tensors = distributed_concat(tensors)
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 168, in distributed_concat
    dist.all_gather(output_tensors, tensor)
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1185, in all_gather
    work = _default_pg.allgather([tensor_list], [tensor])
RuntimeError: All tensor operands to scatter/gather must have the same size
  1%|▉                                                                                                                                                                 | 500/91677 [02:39<8:03:56,  3.14it/s]
Traceback (most recent call last):
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/torch/distributed/launch.py", line 261, in <module>
    main()
  File "/home/caoyq/.conda/envs/torch_cp37/lib/python3.7/site-packages/torch/distributed/launch.py", line 257, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/home/caoyq/.conda/envs/torch_cp37/bin/python', '-u', 'base-Trainer.py', '--local_rank=1', '--mode', 'Data', '--train_data_path', 'utils/dev-small.json', '--output_dir', 'outputs/Data_only_load_data/', '--do_train', '--per_device_train_batch_size', '4', '--save_steps', '100000', '--log_on_each_node', '0']' returned non-zero exit status 1.<|||||>Please use the [forums](https://discuss.huggingface.co/) to debug your code (which you should provide if you want people to be able to help you) as we keep issues for feature requests and identified bugs in the library.<|||||>Met the same problem.  Have you fixed it? @CaoYiqingT <|||||>@Jun-jie-Huang This problem happens when using the log service. I just close the log, the the Trainer runs well. But this is only a temporary measure. Wish it can help you.<|||||>@CaoYiqingT Thanks for your quick response! Closing the log works for me. 👍   <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.