Thanks a lot @ptrblck! I think normalization might be it. That seems to be the cause for inconsistency.
One other inconsistency is that I am unable to run it on a GPU. I did all the usual checks and made sure my device (Windows, GTX GeForce 1060) is running standard codes. There must be something wrong in here where I am changing the device back to CPU or something like that. I am unable to get it run on a GPU. Could you please have a look at my code and check if I am doing something wrong in terms on .to(device) commands