sezer:

If I were to define a prior distribution over class labels, I would write something like exp(w_k * x)*q_k / \sum_j (exp(w_j * x)*q_j) where x is the input, w_k is the parameter vector of class k, and q_k is the weight of class k.


I am a bit lost about this prior distribution concept.
However, I have some counterpoints with your formulation:


when you multiply q_k in the numerator and take log (for CE loss), q_k will be simply ignored as log q_k is a constant.


By trying to multiply q_x with exp() terms (i.e., exp(w_j * x)*q_j), it seems like you are trying to add a constant bias (i.e., exp(w_j * x)*q_j = exp(w_j * x)*exp(log q_j) = exp(w_j * x + log q_j). The network might learn to pump up the logits to avoid this bias (I am not sure!).


What do you think?