Hi @zyyupup,
Can you try this please:
python -m torch.distributed.launch --nproc_per_node=2 main.py -a resnet50 --dali_cpu --fp16 --b 32 --static-loss-scale 128.0 --workers 4 --lr=0.4 ./ 2>&1

And give us full repro step and your environment (Anaconda, PyTorch, DALI versions)