That is the correct understanding of the two numbers.
That is also correct, with float precision the two versions agreed for a few batches up to 32 decimal places when I was not using double precision.  and yes when I switch to double it seems that they agree to 32 decimal places when I specify exact values, but then as soon as any operations are done, forward pass, backward grad or loss calculations, they disagree immediately.
Does not work at all in 1.3.1 means that the built in pytorch functions work properly and and it will converge to a slightly different local minima without using my custom functions, but when it uses my custom functions it makes no difference at all while it used to in 0.3.  The old version is 0.3.0.post4.
It looks like in 1.3.1 it actually is not giving consistent values to begin with.  When I run the same code initialized with the same values I can print the network weights, network output, loss value all to be exactly the same to 16 decimal points but then actual grads that get calculated only agree up to 7.  I can try to make a minimal example to show code if that would help.  My understanding is that floats might lose some precision, but the same math on the same system should always do the same rounding and give the same values.
Here is what I’m doing for debugging now, the second grad value changes every time past the first 7 digits while the first manually calculated grad is identical every time to all digits printed:
(Pdb) X = outputs
(Pdb) y = targets
(Pdb) m = y.shape[0]
(Pdb) grad = F.softmax(X,1)
(Pdb) grad[range(m),y] -= 1
(Pdb) grad = grad/m
(Pdb) ((net.layer1.module.layer1(inputs)).sigmoid()[:,0,0,0] * grad[:,0]).sum().tolist()
0.03555078059434891
(Pdb) (net.layer2.module.layer1.weight.grad[0][0][0][0]).tolist()
0.03555076941847801

Edit:  I noticed a pattern while running this many times and decided to see if there was one.  It turns out after 225 trials this second value only ever takes on 8 different values.  It is possible somewhere in the pytorch code for back propagating cross entropy loss there are 3 bits that are not being set to any particular values so they get randomly assigned?
Eidt2: This seems to be based on batch size.  8 Values was with batch size of 128, with batch size 64 there are only 7 values, with batch size 16 down to 3, batch sizes 4, 2 and 1, consistently return the same correct value