Could you use the latest nightly, remove mixed-precision training (as it seems to also occur using float32), and rerun the script, please?
How long does a usual run take to yield the first NaN output?