Answers.Body: <p>This seems pretty reasonable to me. You can optimize for any function that is <em>proportionate</em> to the negative log-likelihood. Conventionally, we assume that the likelihood of piece of data under a linear model is proportionate to some sort of gaussian function of the difference between the predicted value and the observed value. If you're a Bayesian you'd say this is a probability. If you're a hardcore frequentist, you might quibble about that, but it's still a number between 0 and 1.</p>
  
  <p>If you take the log of this likelihood function however, you'll get a quadratic function of the difference, with some scalers that you don't need to worry about. So you ought to minimize:</p>
  
  <p>$$\sum_y (y-\hat{y})^2$$</p>
  
  <p>This is not a probability, but since it is proportionate to the log of the original likelihood function (which was, in some sense), minimizing it will also minimize the original function.</p>
  
  <p>Hope that helps!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-02T07:16:06.907Z
Answers.Text: Thanks for the help. It made some sense to me but still, I am not able to understand completely. It would be nice if you could provide some sources/supplements that would help me in having a better understanding of your answer.
Answers.UserDisplayName: Akhilesh Pandey
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-02T11:32:26.457Z
Answers.Text: This is a reasonable starting place: https://en.wikipedia.org/wiki/Least_squares#Solving_the_least_squares_problem
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-01T11:55:52.087Z
Answers.LastActivityDate: 2018-09-01T11:55:52.087Z
Answers.UserDisplayName: John Doucette
Body: <p>I am trying to train a supervised model where the output from the model is output of a linear function <span class="math-container">$WX + b$</span>. Kindly note that I'm not using any softmax or <span class="math-container">$\log$</span> softmax on the result of the linear. I am using negative log-likelihood loss function, which takes the input as the linear output from the model and the true labels. I am getting decent accuracy by doing this, but I have read that the input to negative log-likelihood function must be probabilities. Am I doing something wrong?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-30T14:06:20.700Z
Comments.Text: log softmax is used for classification problems only. Are you solving a classification problem? If not you should use a MSEloss or something for regression task. Apart from that softmax takes the network output and transforms it to probabilities. NLL just takes the negative logarithm. So for classification you usually combine a softmax followed by NLL. However, in praxis you often do this in one go that avoids explicit calculation of steps inbetween for better numerical stability, such combined function is often called CrossEntropy loss. So take that for a classifica. task with raw net output.
Comments.UserDisplayName: Marcel_marcel1991
Tags: <deep-learning><objective-functions><activation-functions>
Title: Should the input to the negative log likelihood loss function be probabilities?
UserDisplayName: Akhilesh Pandey
ThreadText: # Should the input to the negative log likelihood loss function be probabilities?
  
  (Asked by: username_0 on 2018-09-01)
  
  I am trying to train a supervised model where the output from the model is output of a linear function $WX + b$. Kindly note that I'm not using any softmax or $\log$ softmax on the result of the linear. I am using negative log-likelihood loss function, which takes the input as the linear output from the model and the true labels. I am getting decent accuracy by doing this, but I have read that the input to negative log-likelihood function must be probabilities. Am I doing something wrong?
  
  **Tags:** deep-learning, objective-functions, activation-functions
  
  > **Comments:**
  > **username_1**: log softmax is used for classification problems only. Are you solving a classification problem? If not you should use a MSEloss or something for regression task. Apart from that softmax takes the network output and transforms it to probabilities. NLL just takes the negative logarithm. So for classification you usually combine a softmax followed by NLL. However, in praxis you often do this in one go that avoids explicit calculation of steps inbetween for better numerical stability, such combined function is often called CrossEntropy loss. So take that for a classifica. task with raw net output.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-09-01. Score: 1
  
  This seems pretty reasonable to me. You can optimize for any function that is *proportionate* to the negative log-likelihood. Conventionally, we assume that the likelihood of piece of data under a linear model is proportionate to some sort of gaussian function of the difference between the predicted value and the observed value. If you're a Bayesian you'd say this is a probability. If you're a hardcore frequentist, you might quibble about that, but it's still a number between 0 and 1.
  
  If you take the log of this likelihood function however, you'll get a quadratic function of the difference, with some scalers that you don't need to worry about. So you ought to minimize:
  
  $$\sum\_y (y-\hat{y})^2$$
  
  This is not a probability, but since it is proportionate to the log of the original likelihood function (which was, in some sense), minimizing it will also minimize the original function.
  
  Hope that helps!
  
  > **Comments:**
  > **username_0**: Thanks for the help. It made some sense to me but still, I am not able to understand completely. It would be nice if you could provide some sources/supplements that would help me in having a better understanding of your answer.  
  > **username_2**: This is a reasonable starting place: https://en.wikipedia.org/wiki/Least_squares#Solving_the_least_squares_problem  
  
  ---
  
  