Sally_Goldin:

RMSE intuitively captures the overall deviation between an output distribution and the ideal distribution. Cross-entropy loss would be more difficult from them to interpret.


If you have the (hardware/time) bandwidth, it may be a good idea to try training with Cross-entropy loss as the loss function. You may find that this also makes RMSE decrease, possibly even faster than when training with RMSE as the measure of loss.