Setting
torch.backends.cudnn.enabled = False
does decrease the variability between the numbers, but I still see some values different on the order of 1e-5. I understand this difference is small and negligible, but, this error will propagate and explode along multiple layers which is happening in the trained model.