Thank you for your response. Acctually I have manually launched this directive “python3 -m torch.distributed …”. It can work on the PBS management system. However, it still fails in the slurm system.  I have tried to use a different version of PyTorch (1.9.0) and the error suggests using torch.distributed.run instead of torch.distributed.launch. And the launch script seems to be simpler than that of before. It may not need to ssh to each node?(can not determine?)Still, that can work on PBS and fails in slurm with the following error
  1 [INFO] 2021-07-10 16:51:24,635 run: Running torch.distributed.run with args: ['/home/chp/bjiangch/.conda/envs/PyTorch-190/lib/pytho    n3.9/site-packages/torch/distributed/run.py', '--nproc_per_node=2', '--nnodes=1', '--rdzv_id=1050201', '--rdzv_backend=c10d', '--rd    zv_endpoint=gnode09:6818', '/home/chp/bjiangch/zyl/2021_0705/program/eann/']
  2 [INFO] 2021-07-10 16:51:24,641 run: Using nproc_per_node=2.
  3 [INFO] 2021-07-10 16:51:24,641 api: Starting elastic_operator with launch configs:
  4   entrypoint       : /home/chp/bjiangch/zyl/2021_0705/program/eann/
  5   min_nodes        : 1
  6   max_nodes        : 1
  7   nproc_per_node   : 2
  8   run_id           : 1050201
  9   rdzv_backend     : c10d
 10   rdzv_endpoint    : gnode09:6818
 11   rdzv_configs     : {'timeout': 900}
 12   max_restarts     : 3
 13   monitor_interval : 5
 14   log_dir          : None
 15   metrics_cfg      : {}
 16 
 17 terminate called after throwing an instance of 'std::system_error'
 18   what():  Connection reset by peer
 19 Fatal Python error: Aborted
 20 
 21 Thread 0x00002b644a6b7a00 (most recent call first):
 22   File "/home/chp/bjiangch/.conda/envs/PyTorch-190/lib/python3.9/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous    _backend.py", line 103 in _call_store
 23   File "/home/chp/bjiangch/.conda/envs/PyTorch-190/lib/python3.9/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous    _backend.py", line 54 in __init__
 24   File "/home/chp/bjiangch/.conda/envs/PyTorch-190/lib/python3.9/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous    _backend.py", line 206 in create_backend
 25   File "/home/chp/bjiangch/.conda/envs/PyTorch-190/lib/python3.9/site-packages/torch/distributed/elastic/rendezvous/registry.py", l    ine 35 in _create_c10d_handler
 26   File "/home/chp/bjiangch/.conda/envs/PyTorch-190/lib/python3.9/site-packages/torch/distributed/elastic/rendezvous/api.py", line 2    53 in create_handler
 27   File "/home/chp/bjiangch/.conda/envs/PyTorch-190/lib/python3.9/site-packages/torch/distributed/elastic/rendezvous/registry.py", l    ine 64 in get_rendezvous_handler
 28   File "/home/chp/bjiangch/.conda/envs/PyTorch-190/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 214 in laun    ch_agent
 29   File "/home/chp/bjiangch/.conda/envs/PyTorch-190/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__i    nit__.py", line 348 in wrapper
 30   File "/home/chp/bjiangch/.conda/envs/PyTorch-190/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 116 in __ca    ll__
 31   File "/home/chp/bjiangch/.conda/envs/PyTorch-190/lib/python3.9/site-packages/torch/distributed/run.py", line 621 in run
 32   File "/home/chp/bjiangch/.conda/envs/PyTorch-190/lib/python3.9/site-packages/torch/distributed/run.py", line 629 in main
 33   File "/home/chp/bjiangch/.conda/envs/PyTorch-190/lib/python3.9/site-packages/torch/distributed/run.py", line 637 in <module>
 34   File "/home/chp/bjiangch/.conda/envs/PyTorch-190/lib/python3.9/runpy.py", line 87 in _run_code
 35   File "/home/chp/bjiangch/.conda/envs/PyTorch-190/lib/python3.9/runpy.py", line 197 in _run_module_as_main
 36 /var/spool/slurmd/job1050201/slurm_script: line 46: 17544 Aborted                 python -m torch.distributed.run --nproc_per_node=    $NPROC_PER_NODE --nnodes=$SLURM_JOB_NUM_NODES --rdzv_id=$SLURM_JOB_ID --rdzv_backend=c10d --rdzv_endpoint=$MASTER:$MPORT $COMMAND >     out

Next is my script
  1 #!/bin/sh
  2 #SBATCH -J 1e5-N-T=1
  3 #SBATCH -p GPU-V100
  4 #SBATCH --qos=gpujoblimit
  5 ##SBATCH --qos=qos_a100_gpu
  6 #SBATCH --gres=gpu:2
  7 #SBATCH --nodes=1
  8 #SBATCH --ntasks-per-node=2 --cpus-per-task=20
  9 #SBATCH --gres-flags=enforce-binding
 10 #SBATCH -o %x.o%j
 11 #SBATCH -e %x.e%j
 12 echo Running on hosts
 13 echo Time is `date`
 14 echo Directory is $PWD
 15 echo This job runs on the following nodes:
 16 echo $SLURM_JOB_NODELIST
 17 # Your conda environment
 18 conda_env=PyTorch-190
 19 
 20 #ATTENTION! HERE MUSTT BE ONE LINE,OR ERROR!
 21 source ~/.bashrc
 22 
 23 module add cuda/11.1
 24 module add /opt/Modules/python/anaconda3
 25 #module add cudnn/7.6.5.32_cuda10.2
 26 conda activate $conda_env
 27 cd $PWD
 28 
 29 
 30 #Number of processes per node to launch (20 for CPU, 2 for GPU)
 31 NPROC_PER_NODE=2
 32 
 33 #The path you place your code
 34 path="/home/chp/bjiangch/zyl/2021_0705/program/eann/"
 35 #This command to run your pytorch script
 36 #You will want to replace this
 37 COMMAND="$path"
 38 
 39 #We want names of master and slave nodes
 40 MASTER=`/bin/hostname -s`
 41 
 42 MPORT=`ss -tan | awk '{print $4}' | cut -d':' -f2 | \
 43       grep "[2-9][0-9]\{3,3\}" | grep -v "[0-9]\{5,5\}" | \
 44       sort | uniq | shuf`
 45 
 46 python -m torch.distributed.run --nproc_per_node=$NPROC_PER_NODE --nnodes=$SLURM_JOB_NUM_NODES --rdzv_id=$SLURM_JOB_ID --rdzv_backe    nd=c10d --rdzv_endpoint=$MASTER:$MPORT $COMMAND >out

Many thanks for your kind help!