NaN values can be created from various sources:

your input to the model might contain invalid values (NaNs and/or Infs) and might thus “poison” your model
your training might diverge, the loss might explode and either directly create invalid loss values or invalid gradients during the backward pass
your gradients might explode due to an “unstable” operation in case you are using custom operations.

You could start by checking the loss and e.g. lowering the learning rate in case you are seeing that the model diverges.