Well, ideally your learning procedure should be robust in the sense that you’d get comparable results from two runs despite the numerical differences.
If you want to have reproducible learning, you need to avoid (or fix!) those functions that introduce the non-linearity. The documentation has a best-knowledge list (I think some particular 2d cross entropy thing is nondeterministic in the forward missing, there is a bug report about that).
If you have one particular function that you want to fix and are willing to dig into that, it’d certainly be interesting, too. For example Chang Rajani in #17501 6 looking at improving the perfomance and eliminating nondeterminism from torch.fold. Most of the time, function use a cuda function atomicAdd which allows multiple threads to add to the same location. As the order isn’t deterministic, this blows up. There are relatively standard techniques to avoid the atomic add (e.g. changing the order of processing things such that only one thread writes to any given gradient value), but it does take a bit of work.
Best regards
Thomas