Can't run QA fine-tune for bert/albert in distributed way
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: HEAD detached at v4.6.1
- Platform: Docker, AWS
- Python version:  Python 3.8.5
- PyTorch version (GPU?): 1.8.0
- Tensorflow version (GPU?):
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [x] the official example scripts: (give details below)
https://github.com/huggingface/transformers/tree/master/examples/pytorch/question-answering
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
Just run:
```
python -m torch.distributed.launch --nproc_per_node=8 run_qa.py \
    --model_name_or_path bert-large-uncased-whole-word-masking \
    --dataset_name squad \
    --do_train \
    --do_eval \
    --learning_rate 3e-5 \
    --num_train_epochs 1 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir ./new_out \
    --max_steps 100 \
    --per_device_eval_batch_size=3   \
    --per_device_train_batch_size=3 \
    --cache_dir .
```
Got error as below:
```
[INFO|trainer.py:2115] 2021-06-14 19:01:08,718 >> ***** Running Evaluation *****
[INFO|trainer.py:2117] 2021-06-14 19:01:08,718 >>   Num examples = 10784
[INFO|trainer.py:2120] 2021-06-14 19:01:08,718 >>   Batch size = 3
Traceback (most recent call last):
  File "run_qa.py", line 622, in <module>
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "run_qa.py", line 622, in <module>
  File "run_qa.py", line 622, in <module>
  File "run_qa.py", line 622, in <module>
  File "run_qa.py", line 622, in <module>
Traceback (most recent call last):
  File "run_qa.py", line 622, in <module>
    main()
  File "run_qa.py", line 581, in main
            main()main()main()


  File "run_qa.py", line 581, in main
  File "run_qa.py", line 581, in main
main()  File "run_qa.py", line 581, in main

  File "run_qa.py", line 581, in main
Traceback (most recent call last):
  File "run_qa.py", line 622, in <module>
    metrics = trainer.evaluate()Traceback (most recent call last):

  File "/workspace/transformers/examples/pytorch/question-answering/trainer_qa.py", line 44, in evaluate
  File "run_qa.py", line 622, in <module>
            metrics = trainer.evaluate()metrics = trainer.evaluate()    output = eval_loop(

main()
  File "/workspace/transformers/examples/pytorch/question-answering/trainer_qa.py", line 44, in evaluate

  File "/workspace/transformers/examples/pytorch/question-answering/trainer_qa.py", line 44, in evaluate
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 2162, in evaluation_loop
      File "run_qa.py", line 581, in main
metrics = trainer.evaluate()
  File "/workspace/transformers/examples/pytorch/question-answering/trainer_qa.py", line 44, in evaluate
    metrics = trainer.evaluate()
    output = eval_loop(  File "/workspace/transformers/examples/pytorch/question-answering/trainer_qa.py", line 44, in evaluate

  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 2162, in evaluation_loop
    output = eval_loop(
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 2162, in evaluation_loop
    main()
output = eval_loop(  File "run_qa.py", line 581, in main

      File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 2162, in evaluation_loop
output = eval_loop(
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 2162, in evaluation_loop
    metrics = trainer.evaluate()
  File "/workspace/transformers/examples/pytorch/question-answering/trainer_qa.py", line 44, in evaluate
    output = eval_loop(
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 2162, in evaluation_loop
    main()
  File "run_qa.py", line 581, in main
    metrics = trainer.evaluate()
  File "/workspace/transformers/examples/pytorch/question-answering/trainer_qa.py", line 44, in evaluate
    output = eval_loop(
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 2162, in evaluation_loop
    logits = self._nested_gather(logits)
    metrics = trainer.evaluate()  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 2252, in _nested_gather

  File "/workspace/transformers/examples/pytorch/question-answering/trainer_qa.py", line 44, in evaluate
        logits = self._nested_gather(logits)output = eval_loop(

logits = self._nested_gather(logits)  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 2252, in _nested_gather
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 2162, in evaluation_loop

  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 2252, in _nested_gather
    logits = self._nested_gather(logits)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 2252, in _nested_gather
    logits = self._nested_gather(logits)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 2252, in _nested_gather
    logits = self._nested_gather(logits)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 2252, in _nested_gather
    logits = self._nested_gather(logits)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 2252, in _nested_gather
    tensors = distributed_concat(tensors)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 154, in distributed_concat
    return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 154, in <genexpr>
        return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)    tensors = distributed_concat(tensors)
tensors = distributed_concat(tensors)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 156, in distributed_concat

  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 154, in distributed_concat
      File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 154, in distributed_concat
logits = self._nested_gather(logits)
      File "/opt/conda/lib/python3.8/site-packages/transformers/trainer.py", line 2252, in _nested_gather
tensors = distributed_concat(tensors)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 154, in distributed_concat
        dist.all_gather(output_tensors, tensor)tensors = distributed_concat(tensors)

      File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1862, in all_gather
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 154, in distributed_concat
return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)
return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 154, in <genexpr>
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 154, in <genexpr>
    return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 154, in <genexpr>
        return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)tensors = distributed_concat(tensors)

    return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)      File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 154, in <genexpr>
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 154, in distributed_concat

tensors = distributed_concat(tensors)return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 156, in distributed_concat


  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 154, in distributed_concat
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 156, in distributed_concat
    return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 156, in distributed_concat
    return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)
return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 156, in distributed_concat

      File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 154, in <genexpr>
dist.all_gather(output_tensors, tensor)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1862, in all_gather
    dist.all_gather(output_tensors, tensor)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1862, in all_gather
    return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 154, in <genexpr>
    dist.all_gather(output_tensors, tensor)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1862, in all_gather
        dist.all_gather(output_tensors, tensor)return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)

  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1862, in all_gather
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 156, in distributed_concat
        work = default_pg.allgather([tensor_list], [tensor])return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)

RuntimeError  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 156, in distributed_concat
: Tensors must be non-overlapping and dense
    dist.all_gather(output_tensors, tensor)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1862, in all_gather
    dist.all_gather(output_tensors, tensor)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1862, in all_gather
    tensors = distributed_concat(tensors)
  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 154, in distributed_concat
    work = default_pg.allgather([tensor_list], [tensor])
    work = default_pg.allgather([tensor_list], [tensor])RuntimeError
: work = default_pg.allgather([tensor_list], [tensor])Tensors must be non-overlapping and dense
RuntimeError
: Tensors must be non-overlapping and dense
RuntimeError:     Tensors must be non-overlapping and densereturn type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)

  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 154, in <genexpr>
    work = default_pg.allgather([tensor_list], [tensor])
    return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)
RuntimeError  File "/opt/conda/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 156, in distributed_concat
: Tensors must be non-overlapping and dense
work = default_pg.allgather([tensor_list], [tensor])
RuntimeError: Tensors must be non-overlapping and dense
    dist.all_gather(output_tensors, tensor)
work = default_pg.allgather([tensor_list], [tensor])  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1862, in all_gather

RuntimeError: Tensors must be non-overlapping and dense
    work = default_pg.allgather([tensor_list], [tensor])
RuntimeError: Tensors must be non-overlapping and dense
Killing subprocess 22340
Killing subprocess 22341
Killing subprocess 22342
Killing subprocess 22343
Killing subprocess 22344
Killing subprocess 22345
Killing subprocess 22346
Killing subprocess 22347
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launch.py", line 340, in <module>
    main()
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launch.py", line 326, in main
    sigkill_handler(signal.SIGTERM, None)  # not coming back
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
    raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', 'run_qa.py', '--local_rank=7', '--model_name_or_path', 'bert-large-uncased-whole-word-masking', '--dataset_name', 'squad', '--do_train', '--do_eval', '--learning_rate', '3e-5', '--num_train_epochs', '1', '--max_seq_length', '384', '--doc_stride', '128', '--output_dir', './new_out', '--max_steps', '100', '--per_device_eval_batch_size=3', '--per_device_train_batch_size=3', '--cache_dir', '.']' returned non-zero exit status 1.
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

@sgugger @philschmid <|||||>Could you confirm #11872 fixes it?<|||||>> Could you confirm #11872 fixes it?

yeah, confirmed, closing issue.