I figured out my problem.
I have a trainable semi-unitary matrix that I iteratively create, and it seems like the iterative process is unstable if the training pushes that matrix too far away from its initial condition. The result is that suddenly the model returns nans even though all weights in the model appear reasonable.