Unnormalized inputs could theoretically create large intermediate activations. If the training diverges due to these high values, you might encounter an overflow and could run into NaNs.
This happened a few times in other posts and you would usually see a very high loss in some iterations before it blows up.