The non-determinism might arise if your batches aren’t sized uniformly?  Without more detail on your training data, it’s just a wild guess.
You might try enabling fp16.  This will give you a lot more breathing room, even if it doesn’t explain the root cause…