Yes, after a single forward pass I get different results to less than 16 digits.  I am not initializing to different random weights or passing in data in a different order.  So I print out every weight in my network and and running them in the diff tool meld, and get exactly the same values up to 16 digits.  But then on the very first forward pass I get losses of 1.9986734118816107 and 1.9986734009655582.  When I was only using float precision things did always stay exactly the same for a few batches before the drift started.  I get what you are saying in the other thread that this will happen and they will converge to equally correct minima.  But I am only debugging with this very simple model, the one I am actually trying to correct once I figure this out is very complicated and uses lots of funky custom functions.  The evidence something is wrong is that with the real model everything works reliably in pytorch 0.3 and does not work at all in 1.3.1.