cudnn.benchmark might yield non-deterministic issue, and I’ve apparently missed these calls in your original script.
These inaccuracies can be accumulated depending on the model depth and architecture.
One way to narrow down this issue would be to compare each layer output using the deterministic setup and your current one.
It would be interesting so see if there is a sudden increase in error, as the difference in the final outputs is large.