@ptrblck, after running dmesg -T | grep xid, it outputs nothing . Will the difference between A100-PCIE-40GB and A100-SXM4-40GB lead to this problem?
And I found some new and interesting phenomena. When I run the following code：
import torch

a = torch.randn(2, 3, device='cuda:0')
print(a)
b = torch.ones(2, 3, device='cuda:1')
print(b)
a1 = a.to('cuda:1')
print(a1)
b1 = b.to('cuda:0')
print(b1)
print(a1 is b)
print(b1 is a)

The output is:
tensor([[-1.7723, -0.0860, -1.6667],
        [ 3.1190, -0.2531, -0.7271]], device='cuda:0')
tensor([[1., 1., 1.],
        [1., 1., 1.]], device='cuda:1')
tensor([[1., 1., 1.],
        [1., 1., 1.]], device='cuda:1')
tensor([[-1.7723, -0.0860, -1.6667],
        [ 3.1190, -0.2531, -0.7271]], device='cuda:0')
False
False

The value of a1 is equal to b and the value of b1 is equal to a. I wonder if this phenomenon can provide new clues.