So I’m still having trouble getting two successive GPU training runs (identical process launches) of only 5 epochs to output exactly the same resulting model in the end. It’s close-ish, but not the same.
If I run the exact same thing on CPU then I get the same outputs to the 16th decimal place, so all good.
If I run the training on the cuda:0 device however, the losses start to differ with time, but only slightly.
I can confirm:

PYTHONHASHSEED is 0 in the environment
All the manual seeding etc from the previous post is happening prior to lazy initialisation of CUDA, and before any model/tensor/training code is reached (there’s some command line argument processing code etc prior to it)
Everything is running strictly in a single python process (no extra data loader workers or whatever)
Prior to training I use torch.rand() to generate a ‘random’ input to the model and evaluate the untrained initial model on this input. Both the input and output tensors are completely deterministic and the same value in each run (judged by printing .mean(), .std(), .median(), .min(), .max() and a fixed element of the tensor to 16 decimal places => all always identical)
After training for 5 epochs I do exactly the same again, and the ‘random’ input torch tensor is identical in each run (i.e. torch.rand() is confirmed to still be perfectly deterministic after training), but the evaluated output tensor differs in the second significant digit and beyond in the printed mean/std/etc values)

As I said, the exact same code running on the cpu device is completely deterministic from start to finish, so what’s the difference when running on CUDA, given that the cuDNN backend should actually be deterministic and with benchmark mode off?