Know this convo is a little old but I’m under the impression there’s some non-determinism in a few cuDNN operations, like atomic adds on floating points? Might be the issue here
https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#reproducibility 65