Training hangs at the very start while using deepspeed
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0
- base docker image: nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04
- Python version: 3.8.8
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): 2.2.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes, using deepspeed

### Who can help
@stas00 for deepspeed 

## Information

Model I am using Layoutlm:

I need to test my layoutlm model by training it only 1 epoch due to test purposes. However, training hangs at the very start without logging anything or returning an error message. When I disable deepspeed and launch my training with `python -m torch.distributed.launch` instead of `deepspeed --num_gpus={torch.cuda.device_count()} --num_nodes=1`, I manage to train for 1 epoch.

The tasks I am working on is:
* Token Classification

## To reproduce

I think it is a general issue. So, training any model with deepspeed for only one epoch may result in hanging process.


## Expected behavior

It would be possible to train a model only for 1 epoch not to waste time while testing. 


Somehow I don't think this has anything to do with how many epochs you're training, at least I have never had a problem training with just one epoch. The problem most likely is elsewhere.

But I can't help you until you give me a way to reproduce your setup.

Ideally please use one of the existing examples, most likely you want this:
https://github.com/huggingface/transformers/tree/master/examples/pytorch/token-classification

1. - launch the example as explain in its README.md w/o deepspeed, than do the same with deepspeed, 
2. - use a public dataset as given in the README.md of the example. 
3. - try a public model first again from the README.md of the example. 

and if it hangs please send the command line you were using after following the above 3 steps.

if it doesn't hang then try: `layoutlm` - then we know it's something specific to that particular model.

Thank you!<|||||>Meanwhile I also tested that layoutlm works with deepspeed. https://github.com/huggingface/transformers/pull/12695<|||||>Thank you @stas00 for your rapid response. I thought that it may be a general issue, that's why I didn't provide any example code. The code now I am working on is a confidential one, I will follow your advice and let you know afterward.<|||||>Also consider using these tools to diagnose the hanging:

- py-spy:
 ```
# trace a running python application - e.g. when it's hanging or very slow and you want to see the backtrace 
pip install py-spy
# dumps traceback for each thread
sudo py-spy dump --pid PID # sudo may or may not be needed
 ```
 - `faulthandler`
```
# make the traceback dumped periodically - every X seconds
import faulthandler
faulthandler.dump_traceback_later(20, repeat=True)
```<|||||>Thank you @stas00 for your suggestions to debug the issue. I have used both tools. FYI: I am using 2 GPUs and, they are stuck while initializing deepspeed. It is not happening every time but so frequently (50 percent of all my tries). Below you can see the outputs.

### Line 414 from `integrations.py`

```
    model, optimizer, _, lr_scheduler = deepspeed.initialize(
        args=SimpleNamespace(**ds_args),  # expects an obj
        model=model,
        model_parameters=model_parameters,
        config_params=config,
    )
```


### This is from `py-spy` for pid 147

```
py-spy dump --pid 147
Process 147: /usr/bin/python -u nlp_ner_layoutlm/train_pipeline/training_step/training_script.py --local_rank=0 --local_example_folder /620a8e1a-2e53-4a9d-8205-61ee86e6453d/layoutlm_data --model_dir /mnt/pipeline/620a8e1a-2e53-4a9d-8205-61ee86e6453d/pytorch_model --batch_size 16 --weight_decay 0.0 --adam_epsilon 1e-08 --learning_rate 2e-05 --epochs 1 --seed 11046060 --tagging_scheme BILOU --profile_logs /mnt/pipeline/620a8e1a-2e53-4a9d-8205-61ee86e6453d/tensorboard_logs --patience 40 --gradient_accumulation_steps 1 --warmup_steps 300 --composite 1 --composite_loss_weight 0.5 --train_dataset_name train --validation_dataset_name validation --use_deepspeed 1 --consolidate 0 --incremental 1 --old_model_dir /mnt/pipeline/620a8e1a-2e53-4a9d-8205-61ee86e6453d/pytorch_model --base_model /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface --recursion_indexes [1, 2] --temperature 2.0
Python v3.8.0 (/usr/bin/python3.8)

Thread 147 (active): "MainThread"
    barrier (torch/distributed/distributed_c10d.py:1967)
    new_group (torch/distributed/distributed_c10d.py:2048)
    _initialize_parameter_parallel_groups (deepspeed/runtime/zero/utils.py:20)
    _configure_distributed_model (deepspeed/runtime/engine.py:578)
    __init__ (deepspeed/runtime/engine.py:149)
    initialize (deepspeed/__init__.py:120)
    init_deepspeed (transformers/integrations.py:414)
    train (composite_trainer.py:168)
    train_model (nlp_ner_layoutlm/layoutlm/utils/training_utils.py:245)
    <module> (training_script.py:65)

```

### This is from `py-spy` for pid 148

```
py-spy dump --pid 148
Process 148: /usr/bin/python -u nlp_ner_layoutlm/train_pipeline/training_step/training_script.py --local_rank=1 --local_example_folder /620a8e1a-2e53-4a9d-8205-61ee86e6453d/layoutlm_data --model_dir /mnt/pipeline/620a8e1a-2e53-4a9d-8205-61ee86e6453d/pytorch_model --batch_size 16 --weight_decay 0.0 --adam_epsilon 1e-08 --learning_rate 2e-05 --epochs 1 --seed 11046060 --tagging_scheme BILOU --profile_logs /mnt/pipeline/620a8e1a-2e53-4a9d-8205-61ee86e6453d/tensorboard_logs --patience 40 --gradient_accumulation_steps 1 --warmup_steps 300 --composite 1 --composite_loss_weight 0.5 --train_dataset_name train --validation_dataset_name validation --use_deepspeed 1 --consolidate 0 --incremental 1 --old_model_dir /mnt/pipeline/620a8e1a-2e53-4a9d-8205-61ee86e6453d/pytorch_model --base_model /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface --recursion_indexes [1, 2] --temperature 2.0
Python v3.8.0 (/usr/bin/python3.8)

Thread 148 (active): "MainThread"
    convert (torch/nn/modules/module.py:610)
    _apply (torch/nn/modules/module.py:381)
    _apply (torch/nn/modules/module.py:359)
    _apply (torch/nn/modules/module.py:359)
    _apply (torch/nn/modules/module.py:359)
    _apply (torch/nn/modules/module.py:359)
    _apply (torch/nn/modules/module.py:359)
    _apply (torch/nn/modules/module.py:359)
    _apply (torch/nn/modules/module.py:359)
    to (torch/nn/modules/module.py:612)
    _configure_distributed_model (deepspeed/runtime/engine.py:575)
    __init__ (deepspeed/runtime/engine.py:149)
    initialize (deepspeed/__init__.py:120)
    init_deepspeed (transformers/integrations.py:414)
    train (composite_trainer.py:168)
    train_model (nlp_ner_layoutlm/layoutlm/utils/training_utils.py:245)
    <module> (training_script.py:65)

```

### This is from `faulthandler`, it always logs below lines for every 20 seconds: 

```
Timeout (0:00:20)!
Thread 0x00007f8be3d2b740 (most recent call first):
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/distributed_c10d.py", line 1967 in barrier
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/distributed_c10d.py", line 2048 in new_group
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/zero/utils.py", line 20 in _initialize_parameter_parallel_groups
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 578 in _configure_distributed_model
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 149 in __init__
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/__init__.py", line 120 in initialize
  File "/usr/local/lib/python3.8/dist-packages/transformers/integrations.py", line 414 in init_deepspeed
  File "/app/nlp_ner_layoutlm/layoutlm/trainers/composite_trainer.py", line 168 in train
  File "/app/nlp_ner_layoutlm/layoutlm/utils/training_utils.py", line 245 in train_model
  File "nlp_ner_layoutlm/train_pipeline/training_step/training_script.py", line 65 in <module>
Timeout (0:00:20)!
Thread 0x00007fa11175f740 (most recent call first):
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 610 in convert
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 381 in _apply
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359 in _apply
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359 in _apply
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359 in _apply
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359 in _apply
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359 in _apply
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359 in _apply
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359 in _apply
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 612 in to
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 575 in _configure_distributed_model
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 149 in __init__
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/__init__.py", line 120 in initialize
  File "/usr/local/lib/python3.8/dist-packages/transformers/integrations.py", line 414 in init_deepspeed
  File "/app/nlp_ner_layoutlm/layoutlm/trainers/composite_trainer.py", line 168 in train
  File "/app/nlp_ner_layoutlm/layoutlm/utils/training_utils.py", line 245 in train_model
  File "nlp_ner_layoutlm/train_pipeline/training_step/training_script.py", line 65 in <module>
```


In the meantime, I will try to adapt my code to share here to allow reproducability. 

<|||||>So you have a syncing problem, the 2 gpus run `barrier` which ensures they arrived to the same point, but one of the gpus doesn't, and so the other is stuck waiting for it.

Are you by chance misconfiguring the launch command? Try to hardcode `2` here:
```
deepspeed --num_gpus={torch.cuda.device_count()} --num_nodes=1
```
could `{torch.cuda.device_count()` be returning a different number than 2?

i.e.:
```
deepspeed --num_gpus=2 --num_nodes=1
```
<|||||>Thanks @stas00 one more time, I have hard-coded the launch command. My training pipeline contains several training steps, and interestingly, the initial 4 training steps with the same configuration have succeeded but 5th step has hanged for some reason in the same way. I can't reproduce it easily, it happens in different steps in my pipeline.

I am still investigating the issue.<|||||>After hard-coding the `num_gpus`, I have followed 2 different approaches with deepspeed and w/o deepspeed. Later, I triggered 3 new pipelines (each has 6 training steps) per each approach.

all pipelines without deepspeed have succeeded.
2 of 3 pipelines with deepspeed have hanged and 1 of them has succeeded. 

🤷🏻‍♂️<|||||>For 2 days, I am triggering lots of trainings with the distributed setting, they didn't hang until now. I am convinced that my issue is related to `deepspeed`. Maybe `deepspeed` doesn't like my configuration :) But I can't go on debugging without much information about being stuck in barrier. I searched on web but can't find any useful info about it.<|||||>If you're able to reproduce the problem with something I can work with directly, I'd be happy to investigate this with you, @hasansalimkanmaz - perhaps you don't need to show us all of your confidential code but just the part where you start things  - it should be pretty generic. 

I'd start with your full app, and remove all code that appears **after** the hanging, - then you can prune it some more binary search-style reduction until you end up with a few lines of code that hang - then we will fix it quickly and most likely you will already see what the problem may be.<|||||>Thanks @stas00 for your kind help. Currently, I don't have time to dive into this issue as I manage to run in a distributed setting without deepspeed, it is not so urgent for now. On the other hand, I will be working on this issue in the next coming weeks. <|||||>　I have a same problem. I run the Bert-Large pretrain with 4 nodes(32 GPUs). When I was debugging, I found that there seemed to be a problem with the training of the last batch. It seems that some Cuda streams are not synchronized, which seems to be related to the pre-compiled deepspeed transformer kernel. I installed deepspeed with DS_BUILD_OPS=1. <|||||>@HydraQYH, please file an Issue with https://github.com/microsoft/DeepSpeed as this is not an HF integration issue. Thank you!<|||||>> @HydraQYH, please file an Issue with https://github.com/microsoft/DeepSpeed as this is not an HF integration issue. Thank you!

Sorry for taking so long to reply to you, some urgent tasks need to be dealt with before. I have solved this problem. I use a distributed environment to run BERT model pre-training. I have 4 machines, each with 8 GPUs（32GB V100）. I found that when the batch size read by some workers is not equal to the preset train_micro_batch_size_per_gpu, it will hang. Therefore, the problem may be caused by different workers with different batch sizes. This situation usually occurs at the end of an epoch, the data is not enough to fill a batch.<|||||>> @HydraQYH, please file an Issue with https://github.com/microsoft/DeepSpeed as this is not an HF integration issue. Thank you!

Sorry for taking so long to reply to you, some urgent tasks need to be dealt with before. I have solved this problem. I use a distributed environment to run BERT model pre-training. I have 4 machines, each with 8 GPUs（32GB V100）. I found that when the batch size read by some workers is not equal to the preset train_micro_batch_size_per_gpu, it will hang. Therefore, the problem may be caused by different workers with different batch sizes. This situation usually occurs at the end of an epoch, the data is not enough to fill a batch.<|||||>Great to hear you have found the culprit, @HydraQYH!

By your description of it, a normal DDP would have had the same problem.

Do you have a solution on your side, or should `transformers` handle such circumstances? Note, that the Deepspeed integration doesn't touch on dataloading, and therefore it's a domain of `transformers` and not of Deepspeed.<|||||>@HydraQYH if you are done with the issue, Could you share the solution? <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.