If it gives Nan's in the gradients then your parameters would become Nan, and you’d see Nan in the loss in the next iteration as well. I take that  “it starts giving NaNs in the gradients. The loss value isn’t nan, however.” means you checked the gradients manually and saw the loss wasn’t Nan but the gradients were?