Basically your model is diverging. Some loss is NaN and it “infects” the weights once it’s backpropagated.
I would bet your loss is increasing rather than decreasing.
If that’s the case try to reduce ur LR