@ptrblck Thanks for your help! Here are outputs:
(pytorch-env) wfang@Precision-5820-Tower-X-Series:~/tempdir$ NCCL_DEBUG=INFO python -m torch.distributed.launch --nproc_per_node=2 w1.py
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
Namespace(local_rank=1)
Namespace(local_rank=0)
1 2 1
0 2 0
gpu 0
Precision-5820-Tower-X-Series:2205681:2205681 [0] NCCL INFO Bootstrap : Using [0]eno1:162.105.95.179<0>
Precision-5820-Tower-X-Series:2205681:2205681 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation

Precision-5820-Tower-X-Series:2205681:2205681 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
Precision-5820-Tower-X-Series:2205681:2205681 [0] NCCL INFO NET/Socket : Using [0]eno1:162.105.95.179<0>
Precision-5820-Tower-X-Series:2205681:2205681 [0] NCCL INFO Using network Socket
NCCL version 2.7.8+cuda11.1
gpu 1
Precision-5820-Tower-X-Series:2205682:2205682 [0] NCCL INFO Bootstrap : Using [0]eno1:162.105.95.179<0>
Precision-5820-Tower-X-Series:2205682:2205682 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation

Precision-5820-Tower-X-Series:2205682:2205682 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
Precision-5820-Tower-X-Series:2205682:2205682 [0] NCCL INFO NET/Socket : Using [0]eno1:162.105.95.179<0>
Precision-5820-Tower-X-Series:2205682:2205682 [0] NCCL INFO Using network Socket

Precision-5820-Tower-X-Series:2205682:2205699 [0] init.cc:573 NCCL WARN Duplicate GPU detected : rank 1 and rank 0 both on CUDA device 17000

Precision-5820-Tower-X-Series:2205681:2205698 [0] init.cc:573 NCCL WARN Duplicate GPU detected : rank 0 and rank 1 both on CUDA device 17000
Precision-5820-Tower-X-Series:2205682:2205699 [0] NCCL INFO init.cc:840 -> 5
Precision-5820-Tower-X-Series:2205681:2205698 [0] NCCL INFO init.cc:840 -> 5
Precision-5820-Tower-X-Series:2205682:2205699 [0] NCCL INFO group.cc:73 -> 5 [Async thread]
Precision-5820-Tower-X-Series:2205681:2205698 [0] NCCL INFO group.cc:73 -> 5 [Async thread]
Traceback (most recent call last):
  File "/home/wfang/tempdir/w1.py", line 28, in <module>
Traceback (most recent call last):
  File "/home/wfang/tempdir/w1.py", line 28, in <module>
    net = distributed_training_init(net)
  File "/home/wfang/tempdir/w1.py", line 18, in distributed_training_init
    net = distributed_training_init(net)
  File "/home/wfang/tempdir/w1.py", line 18, in distributed_training_init
    model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[gpu])
    model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[gpu])  File "/home/wfang/anaconda3/envs/pytorch-env/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 446, in __init__

  File "/home/wfang/anaconda3/envs/pytorch-env/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 446, in __init__
    self._sync_params_and_buffers(authoritative_rank=0)
  File "/home/wfang/anaconda3/envs/pytorch-env/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 457, in _sync_params_and_buffers
    self._sync_params_and_buffers(authoritative_rank=0)
  File "/home/wfang/anaconda3/envs/pytorch-env/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 457, in _sync_params_and_buffers
    self._distributed_broadcast_coalesced(
  File "/home/wfang/anaconda3/envs/pytorch-env/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1155, in _distributed_broadcast_coalesced
    self._distributed_broadcast_coalesced(
  File "/home/wfang/anaconda3/envs/pytorch-env/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1155, in _distributed_broadcast_coalesced
    dist._broadcast_coalesced(
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1614378073850/work/torch/lib/c10d/ProcessGroupNCCL.cpp:825, invalid usage, NCCL version 2.7.8
ncclInvalidUsage: This usually reflects invalid usage of NCCL library (such as too many async ops, too many collectives at once, mixing streams in a group, etc).
    dist._broadcast_coalesced(
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1614378073850/work/torch/lib/c10d/ProcessGroupNCCL.cpp:825, invalid usage, NCCL version 2.7.8
ncclInvalidUsage: This usually reflects invalid usage of NCCL library (such as too many async ops, too many collectives at once, mixing streams in a group, etc).
Killing subprocess 2205681
Killing subprocess 2205682
Traceback (most recent call last):
  File "/home/wfang/anaconda3/envs/pytorch-env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/wfang/anaconda3/envs/pytorch-env/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/wfang/anaconda3/envs/pytorch-env/lib/python3.9/site-packages/torch/distributed/launch.py", line 340, in <module>
    main()
  File "/home/wfang/anaconda3/envs/pytorch-env/lib/python3.9/site-packages/torch/distributed/launch.py", line 326, in main
    sigkill_handler(signal.SIGTERM, None)  # not coming back
  File "/home/wfang/anaconda3/envs/pytorch-env/lib/python3.9/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
    raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/wfang/anaconda3/envs/pytorch-env/bin/python', '-u', 'w1.py', '--local_rank=1']' returned non-zero exit status 1.