Is this specifically a problem with Adam or do other, simpler optimisers such as SGD + momentum give a mismatch in results? Is the behaviour the same on CPU only? Although I don’t think there should be an issue with cuDNN and the RNN module, adding torch.backend.cudnn.enabled = False to the top of your PyTorch code should disable cuDNN and hence give you a sanity check on that.