NCLL No space left on device Error while training with deepspeed
## Environment info

- `transformers` version: 4.4.0
- Platform: docker
- Python version: 3.8
- PyTorch version (GPU?): pip install torch==1.7.1+cu101 torchvision==0.8.2+cu101 -f https://download.pytorch.org/whl/torch_stable.html
- Using distributed or parallel set-up in script?: DDP

Models:

- LayoutLMForTokenClassification

Library:

- deepspeed: @stas00

The problem arises when using:
I add my deepspeed config file to `TrainingArguments` when initialising args object.
During the training, I got a strange error from NCLL backend. Error message is `No space left on device` but it is not possible. 

Here is complete traceback.

```
PyTorch version 1.7.1+cu101 available.
TensorFlow version 2.2.1 available.
Successfully imported onnx version 1.7.0
2021-05-10 10:56:19 DEBUG    tensorflow Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client.
2021-05-10 10:56:24 INFO     __main__   Training in distributed mode...
[2021-05-10 10:56:26,344] [WARNING] [runner.py:122:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2021-05-10 10:56:26,360] [INFO] [runner.py:360:main] cmd = /usr/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMV19 --master_addr=127.0.0.1 --master_port=29500 nlp_ner_layoutlm/train_pipeline/training_step/training_script.py --local_example_folder /f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/layoutlm_data --model_dir /mnt/pipeline/f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/pytorch_model --window_length 512 --batch_size 8 --weight_decay 0.0 --adam_epsilon 1e-08 --learning_rate 2e-05 --epochs 200 --seed 11046060 --bit_precision_fp16 1 --tagging_scheme BILOU --profile_logs /mnt/pipeline/f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/tensorboard_logs --patience 50 --gradient_accumulation_steps 2 --warmup_steps 300 --composite 0 --n_transformer_layers 1 --composite_loss_weight 0.5 --self_training 0 --base_model /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface
[2021-05-10 10:56:28,227] [INFO] [launch.py:73:main] 0 NCCL_DEBUG INFO
[2021-05-10 10:56:28,227] [INFO] [launch.py:73:main] 0 NCCL_VERSION 2.7.8
[2021-05-10 10:56:28,227] [INFO] [launch.py:80:main] WORLD INFO DICT: {'localhost': [0, 1]}
[2021-05-10 10:56:28,227] [INFO] [launch.py:86:main] nnodes=1, num_local_procs=2, node_rank=0
[2021-05-10 10:56:28,227] [INFO] [launch.py:101:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1]})
[2021-05-10 10:56:28,227] [INFO] [launch.py:102:main] dist_world_size=2
[2021-05-10 10:56:28,227] [INFO] [launch.py:104:main] Setting CUDA_VISIBLE_DEVICES=0,1
2021-05-10 10:56:30 DEBUG    tensorflow Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client.
2021-05-10 10:56:30 DEBUG    tensorflow Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client.
PyTorch version 1.7.1+cu101 available.
PyTorch version 1.7.1+cu101 available.
TensorFlow version 2.2.1 available.
TensorFlow version 2.2.1 available.
Successfully imported onnx version 1.7.0
Successfully imported onnx version 1.7.0
2021-05-10 10:56:32 INFO     common_utils.utils Received the following cli arguments: ['nlp_ner_layoutlm/train_pipeline/training_step/training_script.py', '--local_rank=0', '--local_example_folder', '/f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/layoutlm_data', '--model_dir', '/mnt/pipeline/f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/pytorch_model', '--window_length', '512', '--batch_size', '8', '--weight_decay', '0.0', '--adam_epsilon', '1e-08', '--learning_rate', '2e-05', '--epochs', '200', '--seed', '11046060', '--bit_precision_fp16', '1', '--tagging_scheme', 'BILOU', '--profile_logs', '/mnt/pipeline/f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/tensorboard_logs', '--patience', '50', '--gradient_accumulation_steps', '2', '--warmup_steps', '300', '--composite', '0', '--n_transformer_layers', '1', '--composite_loss_weight', '0.5', '--self_training', '0', '--base_model', '/mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface']
2021-05-10 10:56:32 INFO     common_utils.utils Received the following cli arguments: ['nlp_ner_layoutlm/train_pipeline/training_step/training_script.py', '--local_rank=1', '--local_example_folder', '/f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/layoutlm_data', '--model_dir', '/mnt/pipeline/f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/pytorch_model', '--window_length', '512', '--batch_size', '8', '--weight_decay', '0.0', '--adam_epsilon', '1e-08', '--learning_rate', '2e-05', '--epochs', '200', '--seed', '11046060', '--bit_precision_fp16', '1', '--tagging_scheme', 'BILOU', '--profile_logs', '/mnt/pipeline/f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/tensorboard_logs', '--patience', '50', '--gradient_accumulation_steps', '2', '--warmup_steps', '300', '--composite', '0', '--n_transformer_layers', '1', '--composite_loss_weight', '0.5', '--self_training', '0', '--base_model', '/mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface']
2021-05-10 10:56:32 INFO     common_utils.utils Parsed the following parameters: {'local_example_folder': '/f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/layoutlm_data', 'model_dir': '/mnt/pipeline/f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/pytorch_model', 'window_length': 512, 'batch_size': 8, 'weight_decay': 0.0, 'adam_epsilon': 1e-08, 'learning_rate': 2e-05, 'epochs': 200, 'seed': 11046060, 'bit_precision_fp16': 1, 'tagging_scheme': 'BILOU', 'profile_logs': '/mnt/pipeline/f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/tensorboard_logs', 'patience': 50, 'base_model': '/mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface', 'gradient_accumulation_steps': 2, 'warmup_steps': 300, 'old_model_dir': None, 'local_rank': 0, 'sampling_lambda': 0.0, 'self_training': 0, 'composite': 0, 'n_transformer_layers': 1, 'composite_loss_weight': 0.5}
2021-05-10 10:56:32 INFO     common_utils.utils Parsed the following parameters: {'local_example_folder': '/f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/layoutlm_data', 'model_dir': '/mnt/pipeline/f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/pytorch_model', 'window_length': 512, 'batch_size': 8, 'weight_decay': 0.0, 'adam_epsilon': 1e-08, 'learning_rate': 2e-05, 'epochs': 200, 'seed': 11046060, 'bit_precision_fp16': 1, 'tagging_scheme': 'BILOU', 'profile_logs': '/mnt/pipeline/f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/tensorboard_logs', 'patience': 50, 'base_model': '/mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface', 'gradient_accumulation_steps': 2, 'warmup_steps': 300, 'old_model_dir': None, 'local_rank': 1, 'sampling_lambda': 0.0, 'self_training': 0, 'composite': 0, 'n_transformer_layers': 1, 'composite_loss_weight': 0.5}
Didn't find file /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface/added_tokens.json. We won't load it.
Didn't find file /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface/added_tokens.json. We won't load it.
Didn't find file /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface/tokenizer.json. We won't load it.
Didn't find file /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface/tokenizer.json. We won't load it.
loading file /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface/vocab.txt
loading file /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface/vocab.txt
loading file None
loading file None
loading file /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface/special_tokens_map.json
loading file /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface/special_tokens_map.json
loading file /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface/tokenizer_config.json
loading file /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface/tokenizer_config.json
loading file None
loading file None
2021-05-10 10:56:32 INFO     nlp_ner_layoutlm.layoutlm.data_io Creating features from dataset file at /f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/layoutlm_data
2021-05-10 10:56:32 INFO     nlp_ner_layoutlm.layoutlm.data_io Creating features from dataset file at /f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/layoutlm_data
2021-05-10 10:56:35 INFO     nlp_ner_layoutlm.layoutlm.data_io Creating features from dataset file at /f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/layoutlm_data
2021-05-10 10:56:35 INFO     nlp_ner_layoutlm.layoutlm.data_io Creating features from dataset file at /f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/layoutlm_data
2021-05-10 10:56:35 INFO     nlp_ner_layoutlm.layoutlm.trainers Using base model from /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface
2021-05-10 10:56:35 INFO     transformers.configuration_utils loading configuration file /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface/config.json
2021-05-10 10:56:35 INFO     transformers.configuration_utils Model config LayoutLMConfig {
  "attention_probs_dropout_prob": 0.1,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1",
    "2": "LABEL_2",
    "3": "LABEL_3",
    "4": "LABEL_4",
    "5": "LABEL_5",
    "6": "LABEL_6",
    "7": "LABEL_7",
    "8": "LABEL_8",
    "9": "LABEL_9",
    "10": "LABEL_10",
    "11": "LABEL_11",
    "12": "LABEL_12",
    "13": "LABEL_13",
    "14": "LABEL_14",
    "15": "LABEL_15",
    "16": "LABEL_16",
    "17": "LABEL_17",
    "18": "LABEL_18",
    "19": "LABEL_19",
    "20": "LABEL_20",
    "21": "LABEL_21",
    "22": "LABEL_22",
    "23": "LABEL_23",
    "24": "LABEL_24"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1,
    "LABEL_10": 10,
    "LABEL_11": 11,
    "LABEL_12": 12,
    "LABEL_13": 13,
    "LABEL_14": 14,
    "LABEL_15": 15,
    "LABEL_16": 16,
    "LABEL_17": 17,
    "LABEL_18": 18,
    "LABEL_19": 19,
    "LABEL_2": 2,
    "LABEL_20": 20,
    "LABEL_21": 21,
    "LABEL_22": 22,
    "LABEL_23": 23,
    "LABEL_24": 24,
    "LABEL_3": 3,
    "LABEL_4": 4,
    "LABEL_5": 5,
    "LABEL_6": 6,
    "LABEL_7": 7,
    "LABEL_8": 8,
    "LABEL_9": 9
  },
  "layer_norm_eps": 1e-12,
  "max_2d_position_embeddings": 1024,
  "max_position_embeddings": 512,
  "model_type": "layoutlm",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "output_past": true,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "transformers_version": "4.4.0",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 30522
2021-05-10 10:56:35 INFO     nlp_ner_layoutlm.layoutlm.trainers Using base model from /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface
2021-05-10 10:56:35 INFO     transformers.modeling_utils loading weights file /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface/pytorch_model.bin
2021-05-10 10:56:35 INFO     transformers.configuration_utils loading configuration file /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface/config.json
2021-05-10 10:56:35 INFO     transformers.configuration_utils Model config LayoutLMConfig {
  "attention_probs_dropout_prob": 0.1,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1",
    "2": "LABEL_2",
    "3": "LABEL_3",
    "4": "LABEL_4",
    "5": "LABEL_5",
    "6": "LABEL_6",
    "7": "LABEL_7",
    "8": "LABEL_8",
    "9": "LABEL_9",
    "10": "LABEL_10",
    "11": "LABEL_11",
    "12": "LABEL_12",
    "13": "LABEL_13",
    "14": "LABEL_14",
    "15": "LABEL_15",
    "16": "LABEL_16",
    "17": "LABEL_17",
    "18": "LABEL_18",
    "19": "LABEL_19",
    "20": "LABEL_20",
    "21": "LABEL_21",
    "22": "LABEL_22",
    "23": "LABEL_23",
    "24": "LABEL_24"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1,
    "LABEL_10": 10,
    "LABEL_11": 11,
    "LABEL_12": 12,
    "LABEL_13": 13,
    "LABEL_14": 14,
    "LABEL_15": 15,
    "LABEL_16": 16,
    "LABEL_17": 17,
    "LABEL_18": 18,
    "LABEL_19": 19,
    "LABEL_2": 2,
    "LABEL_20": 20,
    "LABEL_21": 21,
    "LABEL_22": 22,
    "LABEL_23": 23,
    "LABEL_24": 24,
    "LABEL_3": 3,
    "LABEL_4": 4,
    "LABEL_5": 5,
    "LABEL_6": 6,
    "LABEL_7": 7,
    "LABEL_8": 8,
    "LABEL_9": 9
  },
  "layer_norm_eps": 1e-12,
  "max_2d_position_embeddings": 1024,
  "max_position_embeddings": 512,
  "model_type": "layoutlm",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "output_past": true,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "transformers_version": "4.4.0",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 30522
2021-05-10 10:56:35 INFO     transformers.modeling_utils loading weights file /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface/pytorch_model.bin
2021-05-10 10:56:50 INFO     transformers.modeling_utils All model checkpoint weights were used when initializing LayoutLMModel.
2021-05-10 10:56:50 INFO     transformers.modeling_utils All the weights of LayoutLMModel were initialized from the model checkpoint at /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface.
If your task is similar to the task the model of the checkpoint was trained on, you can already use LayoutLMModel for predictions without further training.
2021-05-10 10:56:50 INFO     transformers.modeling_utils All model checkpoint weights were used when initializing LayoutLMModel.
2021-05-10 10:56:50 INFO     transformers.modeling_utils All the weights of LayoutLMModel were initialized from the model checkpoint at /mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface.
If your task is similar to the task the model of the checkpoint was trained on, you can already use LayoutLMModel for predictions without further training.
2021-05-10 10:56:53 INFO     nlp_ner_layoutlm.layoutlm.trainers training on cuda
2021-05-10 10:56:53 INFO     nlp_ner_layoutlm.layoutlm.trainers training on cuda
2021-05-10 10:56:56 INFO     transformers.training_args PyTorch: setting up devices
2021-05-10 10:56:56 INFO     transformers.training_args PyTorch: setting up devices
[2021-05-10 10:56:56,398] [INFO] [distributed.py:46:init_distributed] Initializing torch distributed with backend: nccl
[2021-05-10 10:56:56,429] [INFO] [distributed.py:46:init_distributed] Initializing torch distributed with backend: nccl
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:97 [0] NCCL INFO Bootstrap : Using [0]eth0:10.1.0.194<0>
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:97 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:97 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:97 [0] NCCL INFO NET/Socket : Using [0]eth0:10.1.0.194<0>
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:97 [0] NCCL INFO Using network Socket
NCCL version 2.7.8+cuda10.1
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:98 [1] NCCL INFO Bootstrap : Using [0]eth0:10.1.0.194<0>
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:98 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:98 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:98 [1] NCCL INFO NET/Socket : Using [0]eth0:10.1.0.194<0>
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:98 [1] NCCL INFO Using network Socket
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:176 [0] NCCL INFO Channel 00/02 :    0   1
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:177 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:176 [0] NCCL INFO Channel 01/02 :    0   1
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:177 [1] NCCL INFO Trees [0] -1/-1/-1->1->0|0->1->-1/-1/-1 [1] -1/-1/-1->1->0|0->1->-1/-1/-1
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:177 [1] NCCL INFO Setting affinity for GPU 1 to 0fff
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:176 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:176 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1|-1->0->1/-1/-1 [1] 1/-1/-1->0->-1|-1->0->1/-1/-1
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:176 [0] NCCL INFO Setting affinity for GPU 0 to 0fff
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:176 [0] NCCL INFO Channel 00 : 0[100000] -> 1[200000] via direct shared memory
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:177 [1] NCCL INFO Channel 00 : 1[200000] -> 0[100000] via direct shared memory
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:176 [0] NCCL INFO Channel 01 : 0[100000] -> 1[200000] via direct shared memory
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:177 [1] NCCL INFO Channel 01 : 1[200000] -> 0[100000] via direct shared memory
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:176 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:176 [0] NCCL INFO comm 0x7ff1ec001060 rank 0 nranks 2 cudaDev 0 busId 100000 - Init COMPLETE
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:97 [0] NCCL INFO Launch mode Parallel
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:177 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:177 [1] NCCL INFO comm 0x7ff5d8001060 rank 1 nranks 2 cudaDev 1 busId 200000 - Init COMPLETE
2021-05-10 10:56:59 INFO     transformers.trainer Using amp fp16 backend
2021-05-10 10:56:59 INFO     transformers.trainer Using amp fp16 backend
2021-05-10 10:56:59 INFO     nlp_ner_layoutlm.layoutlm.utils_train Starting to train...
2021-05-10 10:56:59 INFO     nlp_ner_layoutlm.layoutlm.utils_train Starting to train...
2021-05-10 10:56:59 INFO     transformers.integrations Keeping the `fp16` config from nlp_ner_layoutlm/toplevel_configs/ds_config.json intact, ignoring any fp16-specific cl args
[2021-05-10 10:56:59,844] [WARNING] [config.py:79:_sanity_check] DeepSpeedConfig: cpu_offload is deprecated. Please use offload_optimizer.
[2021-05-10 10:56:59,891] [INFO] [utils.py:11:_initialize_parameter_parallel_groups] data_parallel_size: 2, parameter_parallel_size: 2
2021-05-10 10:56:59 INFO     transformers.integrations Keeping the `fp16` config from nlp_ner_layoutlm/toplevel_configs/ds_config.json intact, ignoring any fp16-specific cl args
[2021-05-10 10:56:59,932] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed info: version=0.3.16, git-hash=unknown, git-branch=unknown
[2021-05-10 10:56:59,932] [WARNING] [config.py:79:_sanity_check] DeepSpeedConfig: cpu_offload is deprecated. Please use offload_optimizer.
[2021-05-10 10:56:59,954] [INFO] [utils.py:11:_initialize_parameter_parallel_groups] data_parallel_size: 2, parameter_parallel_size: 2
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:183 [0] NCCL INFO Channel 00/02 :    0   1
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:184 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:183 [0] NCCL INFO Channel 01/02 :    0   1
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:184 [1] NCCL INFO Trees [0] -1/-1/-1->1->0|0->1->-1/-1/-1 [1] -1/-1/-1->1->0|0->1->-1/-1/-1
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:184 [1] NCCL INFO Setting affinity for GPU 1 to 0fff
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:183 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:183 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1|-1->0->1/-1/-1 [1] 1/-1/-1->0->-1|-1->0->1/-1/-1
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:183 [0] NCCL INFO Setting affinity for GPU 0 to 0fff
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:183 [0] NCCL INFO Channel 00 : 0[100000] -> 1[200000] via direct shared memory
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:184 [1] NCCL INFO Channel 00 : 1[200000] -> 0[100000] via direct shared memory
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:184 [1] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:184 [1] NCCL INFO include/shm.h:41 -> 2
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:184 [1] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-8f2537dafaac0775-1-0-1 (size 9637888)
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:184 [1] NCCL INFO transport/shm.cc:101 -> 2
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:184 [1] NCCL INFO transport.cc:30 -> 2
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:184 [1] NCCL INFO transport.cc:49 -> 2
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:184 [1] NCCL INFO init.cc:766 -> 2
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:184 [1] NCCL INFO init.cc:840 -> 2
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:98:184 [1] NCCL INFO group.cc:73 -> 2 [Async thread]
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:183 [0] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:183 [0] NCCL INFO include/shm.h:41 -> 2
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:183 [0] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-c43b846667d22574-1-1-0 (size 9637888)
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:183 [0] NCCL INFO transport/shm.cc:101 -> 2
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:183 [0] NCCL INFO transport.cc:30 -> 2
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:183 [0] NCCL INFO transport.cc:49 -> 2
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:183 [0] NCCL INFO init.cc:766 -> 2
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:183 [0] NCCL INFO init.cc:840 -> 2
simple-layoutlm-mmml-1573-investigate-how-to-implement-d-lcbz2t:97:183 [0] NCCL INFO group.cc:73 -> 2 [Async thread]
2021-05-10 10:56:59 ERROR    __main__   NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
Traceback (most recent call last):
  File "nlp_ner_layoutlm/train_pipeline/training_step/training_script.py", line 64, in <module>
    train_model(
  File "/app/nlp_ner_layoutlm/layoutlm/utils_train.py", line 147, in train_model
    raise e
  File "/app/nlp_ner_layoutlm/layoutlm/utils_train.py", line 145, in train_model
    trainer.train()
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 903, in train
    model, optimizer, lr_scheduler = init_deepspeed(self, num_training_steps=max_steps)
  File "/usr/local/lib/python3.8/dist-packages/transformers/integrations.py", line 414, in init_deepspeed
    model, optimizer, _, lr_scheduler = deepspeed.initialize(
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/__init__.py", line 120, in initialize
    engine = DeepSpeedEngine(args=args,
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 149, in __init__
    self._configure_distributed_model(model)
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 591, in _configure_distributed_model
    self._broadcast_model()
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 559, in _broadcast_model
    dist.broadcast(p,
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/distributed_c10d.py", line 864, in broadcast
    work = group.broadcast([tensor], opts)
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
2021-05-10 10:56:59 ERROR    __main__   NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
Traceback (most recent call last):
  File "nlp_ner_layoutlm/train_pipeline/training_step/training_script.py", line 64, in <module>
    train_model(
  File "/app/nlp_ner_layoutlm/layoutlm/utils_train.py", line 147, in train_model
    raise e
  File "/app/nlp_ner_layoutlm/layoutlm/utils_train.py", line 145, in train_model
    trainer.train()
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 903, in train
    model, optimizer, lr_scheduler = init_deepspeed(self, num_training_steps=max_steps)
  File "/usr/local/lib/python3.8/dist-packages/transformers/integrations.py", line 414, in init_deepspeed
    model, optimizer, _, lr_scheduler = deepspeed.initialize(
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/__init__.py", line 120, in initialize
    engine = DeepSpeedEngine(args=args,
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 149, in __init__
    self._configure_distributed_model(model)
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 591, in _configure_distributed_model
    self._broadcast_model()
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 559, in _broadcast_model
    dist.broadcast(p,
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/distributed_c10d.py", line 864, in broadcast
    work = group.broadcast([tensor], opts)
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
Traceback (most recent call last):
  File "nlp_ner_layoutlm/train_pipeline/training_step/training_script.py", line 64, in <module>
    train_model(
  File "/app/nlp_ner_layoutlm/layoutlm/utils_train.py", line 147, in train_model
    raise e
  File "/app/nlp_ner_layoutlm/layoutlm/utils_train.py", line 145, in train_model
    trainer.train()
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 903, in train
    model, optimizer, lr_scheduler = init_deepspeed(self, num_training_steps=max_steps)
  File "/usr/local/lib/python3.8/dist-packages/transformers/integrations.py", line 414, in init_deepspeed
    model, optimizer, _, lr_scheduler = deepspeed.initialize(
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/__init__.py", line 120, in initialize
    engine = DeepSpeedEngine(args=args,
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 149, in __init__
    self._configure_distributed_model(model)
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 591, in _configure_distributed_model
Traceback (most recent call last):
  File "nlp_ner_layoutlm/train_pipeline/training_step/training_script.py", line 64, in <module>
    train_model(
  File "/app/nlp_ner_layoutlm/layoutlm/utils_train.py", line 147, in train_model
    self._broadcast_model()
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 559, in _broadcast_model
    raise e
  File "/app/nlp_ner_layoutlm/layoutlm/utils_train.py", line 145, in train_model
    trainer.train()
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 903, in train
    dist.broadcast(p,
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/distributed_c10d.py", line 864, in broadcast
    work = group.broadcast([tensor], opts)
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
    model, optimizer, lr_scheduler = init_deepspeed(self, num_training_steps=max_steps)
  File "/usr/local/lib/python3.8/dist-packages/transformers/integrations.py", line 414, in init_deepspeed
    model, optimizer, _, lr_scheduler = deepspeed.initialize(
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/__init__.py", line 120, in initialize
    engine = DeepSpeedEngine(args=args,
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 149, in __init__
    self._configure_distributed_model(model)
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 591, in _configure_distributed_model
    self._broadcast_model()
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 559, in _broadcast_model
    dist.broadcast(p,
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/distributed_c10d.py", line 864, in broadcast
    work = group.broadcast([tensor], opts)
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
Killing subprocess 97
Killing subprocess 98
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/launcher/launch.py", line 171, in <module>
    main()
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/launcher/launch.py", line 161, in main
    sigkill_handler(signal.SIGTERM, None)  # not coming back
  File "/usr/local/lib/python3.8/dist-packages/deepspeed/launcher/launch.py", line 139, in sigkill_handler
    raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python', '-u', 'nlp_ner_layoutlm/train_pipeline/training_step/training_script.py', '--local_rank=1', '--local_example_folder', '/f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/layoutlm_data', '--model_dir', '/mnt/pipeline/f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/pytorch_model', '--window_length', '512', '--batch_size', '8', '--weight_decay', '0.0', '--adam_epsilon', '1e-08', '--learning_rate', '2e-05', '--epochs', '200', '--seed', '11046060', '--bit_precision_fp16', '1', '--tagging_scheme', 'BILOU', '--profile_logs', '/mnt/pipeline/f8a83c0e-1438-4a6c-a2d1-f5ed8cf76b0a/tensorboard_logs', '--patience', '50', '--gradient_accumulation_steps', '2', '--warmup_steps', '300', '--composite', '0', '--n_transformer_layers', '1', '--composite_loss_weight', '0.5', '--self_training', '0', '--base_model', '/mnt/pipeline/LAYOUTLM_PRE_TRAINED_MODEL/base-uncased-huggingface']' returned non-zero exit status 1.
2021-05-10 10:57:04 INFO     common_utils.message_utils Loading initial config...
2021-05-10 10:57:04 INFO     common_utils.message_utils Injecting secrets...
2021-05-10 10:57:05 INFO     common_utils.message_utils Done injecting keyvault into config...
2021-05-10 10:57:05 DEBUG    common_utils.kafka Initializing kafka producer
2021-05-10 10:57:05 DEBUG    common_utils.message_utils Sending exception to Kafka...
2021-05-10 10:57:05 DEBUG    common_utils.kafka Message delivered to dev-train-result [1] @ 228
2021-05-10 10:57:05 DEBUG    common_utils.message_utils Exception sent to Kafka.
Traceback (most recent call last):
  File "nlp_ner_layoutlm/train_pipeline/training_step/layoutlm_train_model.py", line 251, in <module>
    run_step(
  File "/app/common_utils/kubeflow_utils.py", line 261, in run_step
    step_callback(**args.__dict__)
  File "nlp_ner_layoutlm/train_pipeline/training_step/layoutlm_train_model.py", line 130, in train_and_save_layoutLM_model
    raise InternalError(
common_utils.errors.InternalError: Something went wrong while training in distributed mode. Process finished with Exit Code 1

```

When I set environment variable to `NCCL_SHM_DISABLE=1`, this error doesn't happen but deepspeed doesn't produce better results and I can't train with bigger batches. 

I am closing the issue, because I see that it is a docker shm issue that is not related to HF. I have solved the issue thanks to [this stackoverflow issue](https://stackoverflow.com/a/46434614/11758585)