Hi, @rvarm1 , I could give you full log of this, but minimal script may need some time to prepare.
And I use NCCL backend.
Use GPU: 2 for training
Use GPU: 5 for training
Use GPU: 4 for training
Use GPU: 6 for training
Use GPU: 0 for training
Use GPU: 3 for training
Use GPU: 7 for training
Use GPU: 1 for training
Namespace(data=(classes=1000; num_training_samples=1281167; input_size=224; transform=(type=normal; color_jit=0.4; dali_pipe=(enable=True; dali_cpu=False)); dataloader=(num_workers=40; sampler=distributed_sampler)); model=(network=GhostNetRE; model_setting=width=0.5,dropout=0.1; model_info=True); trainer=(epochs=360; batch_size=256; dtype=float16; mix_precision_training=True; lr_scheduler=(type=cosine; warmup_lr=0; warmup_epochs=5); optimizer=(learning_rate=2.6; momentum=0.9; weight_decay=3e-05; nesterov=True)); test=(no_test=False; crop_ratio=0.875); tricks=(label_smoothing=(enable=True; smoothing=0.1); no_weight_decay=True; lookahead=False; mixup=(enable=False; alpha=0.2); last_gamma=False; sgd_gc=False); logs=(tensorboard=True; log_interval=200; logging_file_name=distribute_train_imagenet.log); resume=(resume_epoch=0; resume_param=None))
Train with FP16.
Reducer buckets have been rebuilt in this iteration.
Reducer buckets have been rebuilt in this iteration.
Reducer buckets have been rebuilt in this iteration.
Reducer buckets have been rebuilt in this iteration.
Reducer buckets have been rebuilt in this iteration.
Reducer buckets have been rebuilt in this iteration.
Reducer buckets have been rebuilt in this iteration.
Reducer buckets have been rebuilt in this iteration.
Epoch 0, Node 0, GPU 7, Iter 200, Top1 Accuracy:0.0029928, Loss:6.847, 622 samples/s. lr: 0.16806.
Epoch 0, Node 0, GPU 3, Iter 200, Top1 Accuracy:0.0031095, Loss:6.8501, 622 samples/s. lr: 0.16806.
Epoch 0, Node 0, GPU 2, Iter 200, Top1 Accuracy:0.0022932, Loss:6.8461, 622 samples/s. lr: 0.16806.
Epoch 0, Node 0, GPU 6, Iter 200, Top1 Accuracy:0.0026625, Loss:6.8474, 622 samples/s. lr: 0.16806.
Epoch 0, Node 0, GPU 0, Iter 200, Top1 Accuracy:0.0027596, Loss:6.8466, 622 samples/s. lr: 0.16806.
Epoch 0, Node 0, GPU 5, Iter 200, Top1 Accuracy:0.0026819, Loss:6.8472, 622 samples/s. lr: 0.16806.
Epoch 0, Node 0, GPU 4, Iter 200, Top1 Accuracy:0.0026819, Loss:6.8492, 622 samples/s. lr: 0.16806.
Epoch 0, Node 0, GPU 1, Iter 200, Top1 Accuracy:0.0023515, Loss:6.8483, 622 samples/s. lr: 0.16806.
Epoch 0, Node 0, GPU 4, Iter 400, Top1 Accuracy:0.012313, Loss:6.5364, 587 samples/s. lr: 0.33446.
Epoch 0, Node 0, GPU 3, Iter 400, Top1 Accuracy:0.012235, Loss:6.5393, 587 samples/s. lr: 0.33446.
Epoch 0, Node 0, GPU 6, Iter 400, Top1 Accuracy:0.012293, Loss:6.5403, 587 samples/s. lr: 0.33446.
Epoch 0, Node 0, GPU 1, Iter 400, Top1 Accuracy:0.012595, Loss:6.5376, 587 samples/s. lr: 0.33446.
Epoch 0, Node 0, GPU 5, Iter 400, Top1 Accuracy:0.012089, Loss:6.5356, 587 samples/s. lr: 0.33446.
Epoch 0, Node 0, GPU 7, Iter 400, Top1 Accuracy:0.012118, Loss:6.5386, 587 samples/s. lr: 0.33446.
Epoch 0, Node 0, GPU 0, Iter 400, Top1 Accuracy:0.012313, Loss:6.5372, 587 samples/s. lr: 0.33446.
Epoch 0, Node 0, GPU 2, Iter 400, Top1 Accuracy:0.01169, Loss:6.5363, 587 samples/s. lr: 0.33446.
Epoch 0, Node 0, GPU 2, Iter 600, Top1 Accuracy:0.027331, Loss:6.2461, 567 samples/s. lr: 0.50086.
Epoch 0, Node 0, GPU 4, Iter 600, Top1 Accuracy:0.02813, Loss:6.2486, 567 samples/s. lr: 0.50086.
Epoch 0, Node 0, GPU 5, Iter 600, Top1 Accuracy:0.027961, Loss:6.2488, 567 samples/s. lr: 0.50086.
Epoch 0, Node 0, GPU 6, Iter 600, Top1 Accuracy:0.027968, Loss:6.2517, 567 samples/s. lr: 0.50086.
Epoch 0, Node 0, GPU 1, Iter 600, Top1 Accuracy:0.028065, Loss:6.2518, 567 samples/s. lr: 0.50086.
Epoch 0, Node 0, GPU 7, Iter 600, Top1 Accuracy:0.027786, Loss:6.2526, 567 samples/s. lr: 0.50086.
Epoch 0, Node 0, GPU 3, Iter 600, Top1 Accuracy:0.027428, Loss:6.2534, 567 samples/s. lr: 0.50086.
Epoch 0, Node 0, GPU 0, Iter 600, Top1 Accuracy:0.027636, Loss:6.2534, 567 samples/s. lr: 0.50086.
Finish one epoch cost 00:04:43, speed: 4527 samples/s.
rank:3 get second barrier.
rank:4 get second barrier.
rank:1 get second barrier.
rank:6 get second barrier.
rank:7 get second barrier.
rank:5 get second barrier.
rank:2 get second barrier.
Test Epoch 0, Top1 Accuracy:0.08558, Top5 Accuracy:0.22848, Loss:5.3841
rank:0 get second barrier.