InnovArul:

Technically, Cross-entropy (CE) is independent of softmax and a generic concept to measure distances/differences between two probability distributions. It so happens that one of the ways to get probability distributions in deep learning (especially for classification problems) is to use softmax. Ideally, CE loss can be realized by log_softmax + NLL loss. However, for numerical stability reasons, CE loss in pytorch is coupled with Softmax directly.


I understand that, but in this case aren’t they the same? log_softmax + NLL loss should be the same thing as the negative log-likelihood of multinomial logistic regression (which I called softmax regression before). The equation in my previous post shows that the loss is defined as the negative log-likelihood of multinomial logistic regression (I suppose x[class] is class_weights*input).
If I were to define a prior distribution over class labels, I would write something like exp(w_k * x)*q_k / \sum_j (exp(w_j * x)*q_j) where x is the input, w_k is the parameter vector of class k, and q_k is the weight of class k. I think, this way I would get the correctly normalized probabilities considering the weights as well. And I would end up something different than the weighted loss given in [1]. Btw my first attempt (second equation in my first post) seems to be wrong.