Thank you for your reply.
I tested the communication speed by p2pBandwidthLatencyTest, the result is as following:
P2P Connectivity Matrix
     D\D     0     1     2     3     4     5     6     7
     0       1     1     1     1     0     0     0     1
     1       1     1     1     1     0     0     1     0
     2       1     1     1     1     0     1     0     0
     3       1     1     1     1     1     0     0     0
     4       0     0     0     1     1     1     1     1
     5       0     0     1     0     1     1     1     1
     6       0     1     0     0     1     1     1     1
     7       1     0     0     0     1     1     1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3      4      5      6      7
     0 767.74   6.80   6.70   6.76   6.36   6.48   6.51   6.43
     1   6.74 770.33   6.72   6.78   6.40   6.48   6.53   6.44
     2   6.90   6.93 770.53   6.58   6.39   6.48   6.52   6.44
     3   7.02   7.10   6.49 771.49   6.42   6.48   6.52   6.44
     4   6.72   6.70   6.57   6.57 770.70   6.33   6.49   6.40
     5   6.78   6.86   6.69   6.73   6.30 771.35   6.47   6.39
     6   6.70   6.73   6.56   6.64   6.48   6.48 771.28   6.29
     7   6.80   6.82   6.71   6.75   6.58   6.58   6.43 770.33
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2      3      4      5      6      7
     0 769.29  48.48  48.49  24.25   6.71   6.66   6.76  24.26
     1  48.48 771.21  24.25  48.49   6.72   6.68  24.25   6.68
     2  48.48  24.26 770.81  24.25   6.72  48.48   6.74   6.69
     3  24.25  48.48  24.25 771.18  48.49   6.66   6.74   6.69
     4   6.86   6.89   6.70  48.48 770.87  24.25  48.48  24.25
     5   6.84   6.85  48.49   6.75  24.25 771.44  24.25  48.49
     6   6.86  24.25   6.75   6.79  48.48  24.25 771.74  48.48
     7  24.25   6.90   6.75   6.79  24.25  48.48  48.48 771.11
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3      4      5      6      7
     0 769.95   8.07  10.58  10.36  10.25  10.09  10.22   9.92
     1   8.32 771.01  10.50  10.38  10.22  10.12  10.18   9.99
     2  10.48  10.76 770.73   7.88  10.05   9.88  10.29   9.91
     3  10.38  10.68   7.89 772.27  10.01  10.18  10.14  10.17
     4  10.15  10.40  10.22  10.00 772.02   7.57  10.18   9.76
     5  10.03  10.29  10.08  10.20   7.64 772.21   9.99   9.95
     6  10.17  10.30  10.29  10.13  10.05   9.81 770.79   7.56
     7  10.06  10.09  10.03  10.38   9.81  10.07   7.57 771.87
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3      4      5      6      7
     0 770.84  96.91  96.91  48.49  10.37  10.10  10.35  48.49
     1  96.86 771.08  48.49  96.92  10.52  10.29  48.49  10.05
     2  96.85  48.49 769.93  48.50  10.16  96.91  10.32  10.05
     3  48.49  96.91  48.49 769.68  96.90  10.18  10.11  10.40
     4  10.30  10.50  10.14  96.92 771.43  48.49  96.85  48.49
     5   9.97  10.16  96.91  10.18  48.49 771.37  48.49  96.91
     6  10.32  48.49  10.25  10.05  96.85  48.48 771.73  96.91
     7  48.50   9.99   9.92  10.30  48.48  96.91  96.86 771.20

By the way, torch 1.10 can speed up training, but communication overhead is still high.