@ptrblck , I just run the cuda-sample p2pBandwidthLatencyTest 2 which demonstrates Peer-To-Peer (P2P) data transfers between pairs of GPUs and computes latency and bandwidth., and it outputs:
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, A100-PCIE-40GB, pciBusID: 1, pciDeviceID: 0, pciDomainID:0
Device: 1, A100-PCIE-40GB, pciBusID: 24, pciDeviceID: 0, pciDomainID:0
Device: 2, A100-PCIE-40GB, pciBusID: 41, pciDeviceID: 0, pciDomainID:0
Device: 3, A100-PCIE-40GB, pciBusID: 61, pciDeviceID: 0, pciDomainID:0
Device: 4, A100-PCIE-40GB, pciBusID: 81, pciDeviceID: 0, pciDomainID:0
Device: 5, A100-PCIE-40GB, pciBusID: a1, pciDeviceID: 0, pciDomainID:0
Device: 6, A100-PCIE-40GB, pciBusID: c1, pciDeviceID: 0, pciDomainID:0
Device: 7, A100-PCIE-40GB, pciBusID: e1, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=0 CAN Access Peer Device=2
Device=0 CAN Access Peer Device=3
Device=0 CAN Access Peer Device=4
Device=0 CAN Access Peer Device=5
Device=0 CAN Access Peer Device=6
Device=0 CAN Access Peer Device=7
Device=1 CAN Access Peer Device=0
Device=1 CAN Access Peer Device=2
Device=1 CAN Access Peer Device=3
Device=1 CAN Access Peer Device=4
Device=1 CAN Access Peer Device=5
Device=1 CAN Access Peer Device=6
Device=1 CAN Access Peer Device=7
Device=2 CAN Access Peer Device=0
Device=2 CAN Access Peer Device=1
Device=2 CAN Access Peer Device=3
Device=2 CAN Access Peer Device=4
Device=2 CAN Access Peer Device=5
Device=2 CAN Access Peer Device=6
Device=2 CAN Access Peer Device=7
Device=3 CAN Access Peer Device=0
Device=3 CAN Access Peer Device=1
Device=3 CAN Access Peer Device=2
Device=3 CAN Access Peer Device=4
Device=3 CAN Access Peer Device=5
Device=3 CAN Access Peer Device=6
Device=3 CAN Access Peer Device=7
Device=4 CAN Access Peer Device=0
Device=4 CAN Access Peer Device=1
Device=4 CAN Access Peer Device=2
Device=4 CAN Access Peer Device=3
Device=4 CAN Access Peer Device=5
Device=4 CAN Access Peer Device=6
Device=4 CAN Access Peer Device=7
Device=5 CAN Access Peer Device=0
Device=5 CAN Access Peer Device=1
Device=5 CAN Access Peer Device=2
Device=5 CAN Access Peer Device=3
Device=5 CAN Access Peer Device=4
Device=5 CAN Access Peer Device=6
Device=5 CAN Access Peer Device=7
Device=6 CAN Access Peer Device=0
Device=6 CAN Access Peer Device=1
Device=6 CAN Access Peer Device=2
Device=6 CAN Access Peer Device=3
Device=6 CAN Access Peer Device=4
Device=6 CAN Access Peer Device=5
Device=6 CAN Access Peer Device=7
Device=7 CAN Access Peer Device=0
Device=7 CAN Access Peer Device=1
Device=7 CAN Access Peer Device=2
Device=7 CAN Access Peer Device=3
Device=7 CAN Access Peer Device=4
Device=7 CAN Access Peer Device=5
Device=7 CAN Access Peer Device=6

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1     2     3     4     5     6     7
     0       1     1     1     1     1     1     1     1
     1       1     1     1     1     1     1     1     1
     2       1     1     1     1     1     1     1     1
     3       1     1     1     1     1     1     1     1
     4       1     1     1     1     1     1     1     1
     5       1     1     1     1     1     1     1     1
     6       1     1     1     1     1     1     1     1
     7       1     1     1     1     1     1     1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3      4      5      6      7
     0 1154.84  14.60  14.67  14.33  15.10  15.62  15.94  17.48
     1  21.10 1290.26  15.47  15.56  15.70  15.75  16.02  17.27
     2  15.24  15.12 1177.47  14.75  15.01  15.67  15.89  17.55
     3  15.13  14.92  15.05 1167.79  16.10  16.14  16.37  17.47
     4  15.11  15.22  15.12  21.26 1292.39  17.51  18.07  15.18
     5  15.15  15.27  15.11  18.93  21.74 1292.39  18.05  16.26
     6  15.02  15.60  19.99  21.53  21.61  21.34 1291.32  15.86
     7  14.74  17.48  21.25  21.47  21.58  21.34  21.59 1295.61
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2      3      4      5      6      7
     0 1156.55   2.11   2.78   2.78   2.78   2.78   2.78   2.78
     1   1.24 1291.32   1.62   1.63   1.63   1.33   1.62   1.62
     2   2.40   2.78 1293.46   2.78   2.78   2.78   2.78   2.78
     3   2.41   2.17   2.78 1293.46   2.78   2.78   2.78   2.78
     4   2.14   2.44   2.78   2.78 1293.46   2.78   2.78   2.78
     5   2.33   2.17   2.78   2.78   2.78 1291.32   2.78   2.78
     6   2.40   2.44   2.78   2.78   2.44   2.78 1293.46   2.78
     7   2.14   2.78   2.78   2.78   2.78   2.78   2.78 1293.46
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3      4      5      6      7
     0 1180.58  20.51  19.38  19.44  20.11  20.03  20.32  23.85
     1  23.52 526.72  19.13  19.57  19.82  19.82  20.03  29.41
     2  19.88  18.65 1204.24  18.97  19.56  19.41  19.60  29.57
     3  19.53  20.43  19.69 1291.32  20.39  20.29  20.53  23.41
     4  19.79  20.58  21.06  22.56 1306.98  21.44  21.89  20.29
     5  19.89  20.85  20.41  22.24  30.66 1308.08  21.96  21.02
     6  19.60  21.33  19.79  22.55  30.23  30.73 1308.63  20.59
     7  20.16  29.87  21.30  23.69  22.56  25.27  23.09 1306.98
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3      4      5      6      7
     0 1184.16   3.25   5.43   4.79   4.28   4.83   4.80   4.81
     1   3.25 487.44   2.66   3.24   3.25   3.25   1.87   3.24
     2   4.70   3.24 1309.17   4.82   4.28   4.32   4.80   4.83
     3   3.87   3.23   5.57 1304.26   4.28   4.83   4.29   3.87
     4   3.87   2.65   4.23   5.57 1308.63   4.28   5.45   4.81
     5   4.73   2.65   4.77   4.28   3.90 1308.63   4.80   4.24
     6   4.20   3.23   4.79   4.24   4.28   4.28 1309.72   4.79
     7   4.28   2.65   4.29   5.43   4.83   4.79   4.29 1304.80
P2P=Disabled Latency Matrix (us)
   GPU     0      1      2      3      4      5      6      7
     0   4.34  23.57  23.59  23.67  22.00  21.84  22.72  21.99
     1  23.53   2.31  21.76  21.65  15.04  18.91  19.76  18.93
     2  23.57  14.54   2.37  20.29  21.53  15.93  18.08  15.33
     3  23.58  21.56  21.56   2.33  16.20  20.58  21.47  20.71
     4  22.91  15.80  15.57  19.92   2.38  20.55  20.56  18.27
     5  22.50  17.00  19.01  17.91  20.55   2.38  20.58  20.54
     6  22.93  14.91  20.48  12.59  20.56  20.58   2.30  19.28
     7  22.30  21.55  13.68  21.55  21.55  21.55  21.55   2.50

   CPU     0      1      2      3      4      5      6      7
     0   3.79  11.84  12.26  11.93  10.96  10.78  10.57  11.15
     1  12.22   3.54  11.48  11.29  10.21  10.22   9.91  10.45
     2  12.05  11.31   3.59  11.17  10.09  10.12   9.83  10.39
     3  11.93  11.12  11.06   3.53  10.00  10.05   9.78  10.21
     4  11.18  10.38  10.46  10.22   3.24   9.45   9.19   9.56
     5  11.12  10.40  10.50  10.28   9.38   3.15   9.21   9.59
     6  10.95  10.18  10.29  10.10   9.18   9.25   3.08   9.46
     7  11.62  10.76  10.86  10.51   9.54   9.53   9.33   3.77
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1      2      3      4      5      6      7
     0   4.35 49204.95 49204.80 49204.80 49204.78 49204.77 49204.79 49204.79
     1 49205.36   2.31 49204.98 49204.97 49205.17 49204.88 49205.20 49205.15
     2 49204.99 49204.85   2.34 49204.81 49204.84 49204.82 49204.76 49204.83
     3 49205.01 49204.90 49204.84   2.33 49204.84 49204.89 49204.86 49204.79
     4 49204.87 49204.77 49204.73 49204.75   2.39 49204.68 49204.75 49204.73
     5 49204.92 49204.78 49204.81 49204.81 49204.78   2.38 49204.80 49204.76
     6 49205.02 49204.82 49204.87 49204.86 49204.85 49204.83   2.29 49204.85
     7 49205.04 49204.88 49204.90 49204.86 49204.82 49204.83 49204.85   2.47

   CPU     0      1      2      3      4      5      6      7
     0   5.10   2.95   3.02   3.08   3.01   3.03   3.01   2.97
     1   2.98   3.78   2.75   2.85   2.87   2.86   2.86   2.91
     2   2.96   2.77   3.81   2.81   3.05   2.83   2.98   2.72
     3   3.02   2.84   2.77   3.72   2.83   2.86   2.83   2.80
     4   2.82   2.60   2.53   2.62   3.38   2.59   2.63   2.53
     5   2.53   2.56   2.53   2.57   2.43   3.46   2.49   2.38
     6   2.66   2.66   2.55   2.51   2.74   2.67   6.27   4.33
     7   2.70   2.62   2.56   2.58   2.67   2.66   2.57   4.59

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

I wonder if this results can provide new clues.