Oh, sorry, I didn’t get that right.
Well, so the gradient would also be subject to numerical accuracy differences similar to those in the forward. If you want to be certain, you could extend your test program to run torch.autograd.gradcheck.
Those are legitimate and, unfortunate as it is, it would seem that the model training is quite brittle (which is the case in deep learning more often than it should).
Best regards
Thomas