Thanks for the quick response! How could I check whether the amp code is using deterministic cuDNN kernels? I know the non-determinism before using amp came from Conv2d layers preceding another Conv2d layer during backprop, if that narrows it down.