Hi,
I’ve had some trouble in reproducing results, and after reading a few posts there seem to be multiple causes for non-determinism, some which are expected. I’d like to check if I got it right.
Issue 1
Dataloaders with multiple threads: They seem to be problematic, even if randoms seeds are set beforehand. [ref].
Issue 2
cuDNN: Apparently cuDNN seems to have non-deterministic kernels [ref1][ref2]
Issue 3
GPU: Apparently, some reductions are non-deterministic in GPU, even without cuDNN. For instance, floating point addition is not associative, and so the result is affected by the order in which the partial results for each thread are collected affects the result. This is expected behaviour and should not affect the results dramatically [ref 39].
(Note: deterministic GPU reductions seem to have been added to TensorFlow only recently 49)
Is there anything I’m missing/got wrong?