sezer:

But the original form of the cross-entropy loss is exactly the negative log-likelihood of softmax regression:


Technically, Cross-entropy (CE) is independent of softmax and a generic concept to measure distances/differences between two probability distributions. It so happens that one of the ways to get probability distributions in deep learning (especially for classification problems) is to use softmax. Ideally, CE loss can be realized by log_softmax + NLL loss. However, for numerical stability reasons, CE loss in pytorch is coupled with Softmax directly.



 sezer:

The denominator (the log term in the first equation) seems to be weighted because of the logarithm, I think I missed this.


Can you elaborate on this a bit more?