Code:
model0 = nn.Linear(10, 10)
model1 = nn.Linear(10, 10)
model2 = nn.Linear(10, 10)
model3 = nn.Linear(10, 10)
input = torch.rand(128 ,10)
output = model3(model2(model1(model0(input))))
model0.to('cuda:0')
model1.to('cuda:1')
model2.to('cuda:2')
model3.to('cuda:3')
pred = model3(model2(model1(model0(input.to('cuda:0')).to('cuda:1')).to('cuda:2')).to('cuda:3')).cpu()
assert torch.allclose(output, pred) # Return False

How to fix this test? The maximum difference between output and pred is just 6.3330e-08 so that the implementation is not wrong but some system issue, I think.
Is there any possible option to make this thing right?
I don’t want to lose precision accuracy so that adding atoi=1e-7 to torch.allclose is not I want.
Thanks,