@Kiuk_Chung I use shell script from a terminal, and the commad is just like :
CUDA_VISIBLE_DEVICES=6,7 python -m torch.distributed.launch --nproc_per_node 2 --master_port=26500  training.py --arg1 --arg2

This error always appeared after 10 epochs, .