So you mean that you compare the second hook that prints tensor([-5.7899e-17, 1.3624e-16, -1.5690e-16, -1.4070e-16, 2.3213e-16], device=‘cuda:0’, dtype=torch.float64) vs [2.62290190e-07 3.63598041e-07 6.38378239e-08 1.66533454e-08 3.88578059e-08] ?
The difference looks small enough that this is numerical errors no?