@rvarm1 ,
I firstly tried the following 2 commands to start to 2 tasks which include 2 sub-processes respectively. but I encountered the Address already in use issue.
CUDA_VISIBLE_DEVICES=1,3 WORLD_SIZE=2 MASTER_PORT=44144 python -m torch.distributed.launch --nproc_per_node=2 train.py

CUDA_VISIBLE_DEVICES=4,5 WORLD_SIZE=2 MASTER_PORT=44145 python -m torch.distributed.launch --nproc_per_node=2 train.py

I then use the following 2 commands to start 2 tasks. 2 tasks with 2 sub-prcesses are started successfully respectively.
CUDA_VISIBLE_DEVICES=1,3 WORLD_SIZE=2 python -m torch.distributed.launch --nproc_per_node=2 --master_port 47769 train.py

CUDA_VISIBLE_DEVICES=4,5 WORLD_SIZE=2 python -m torch.distributed.launch --nproc_per_node=2 --master_port 47770 train.py