I was doing an unsupervised experiment where some parts of conv weights are masked so does a position in output sees only specific input pixels. While debugging the model all the output positions were depending on input pixels that they shouldn’t(found by backpropagating gradients). I made sure all masking is correct. Finally after hours of debugging when torch.backends.cudnn.deterministic = True was set, all the output pixels were looking only at correct input positions. How is this possible? Does cudnn uses optimizations that may alter the gradients during backprop?