If the loss is getting a NaN value, I assume your model output is also already containing invalid values.
Could you post the model you are using as well as the stats of your input data (min., max., mean) so that we could try to reproduce this issue?