The problem might be, that your loss has no upper bound. So in case your mean has high value, the resulting loss will be extremely large and possibly resulting in an Nan.
You might be able to alleviate this by limiting the loss.
Either by clipping the loss to a maximum value or by limiting the range of the mean values by using a sigmoid prior to entering the loss calculation.