Hi K. Frank!
I am looking at values after single batches.  To try to fix this for the more complicated project I am using a model is very simple, its just a conv layer, a sigmoid, and another conv.  I am printing the exact weights and bias as well as the grad for every weight after each batch as well as ensuring they initialize to the same values.  It gives the same up to 16 decimal places for the first few batches but then drift starts to accumulate.  With double precision the drift started immediately.  Maybe I am not using .double() in the correct places?  I basically just made everywhere I was doing .cuda() to now be .cuda().double().