Answers.Body: <p>Square loss is fine for regression, since minimizing it is the same as maximizing the likelihood of the model parameters (under assumption that the error is Gaussian). However, if the model directly produces probabilities, then it is natural to use these probabilities directly within the loss. Hence, in all classification models we prefer to minimize negative log-likelihood of the correct class.</p>
  
  <p>Note that choosing a natural loss leads to several practical advantages. In particular, applying a quadratic loss after a sigmoid activation leads to very poor gradients when the sigmoid is saturated in the wrong direction. The negative log-likelihood loss has no such problems.</p>
  
  <p>This issue is not specific for neural networks. Logistic regression has used the negative log likelihood loss ever since 1958.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-31T15:02:00.913Z
Answers.Text: Minimize **negative** log-likelihood, or maximize log-likelihood, right? :D
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-31T12:17:15.077Z
Answers.LastActivityDate: 2019-01-31T17:17:14.343Z
Answers.LastEditDate: 2019-01-31T17:17:14.343Z
Answers.UserDisplayName: ssegvic
Body: <p>In LMS(least mean square) since, we use a quadratic error function, and quadratic functions are generally parabola in (some convex like shape). I wonder whether that is the reason why we use least square error metric? If that is not the case(its not ALWAYS convex or reason WHY we use LMS), what is the reason then? why this metric changes for deep learning/neural networks but works for regression problems?</p>
  
  <p>[EDIT]: Will this always be a convex function or is there any possibility that it will not be convex?</p>
  
Tags: <neural-networks><deep-learning><gradient-descent><linear-regression>
Title: Will LMS always be convex function? If yes, then why do we change it for neural networks?
UserDisplayName: codecracker
ThreadText: # Will LMS always be convex function? If yes, then why do we change it for neural networks?
  
  (Asked by: username_0 on 2019-01-30)
  
  In LMS(least mean square) since, we use a quadratic error function, and quadratic functions are generally parabola in (some convex like shape). I wonder whether that is the reason why we use least square error metric? If that is not the case(its not ALWAYS convex or reason WHY we use LMS), what is the reason then? why this metric changes for deep learning/neural networks but works for regression problems?
  
  [EDIT]: Will this always be a convex function or is there any possibility that it will not be convex?
  
  **Tags:** neural-networks, deep-learning, gradient-descent, linear-regression
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-01-31. Score: 2 | Accepted answer
  
  Square loss is fine for regression, since minimizing it is the same as maximizing the likelihood of the model parameters (under assumption that the error is Gaussian). However, if the model directly produces probabilities, then it is natural to use these probabilities directly within the loss. Hence, in all classification models we prefer to minimize negative log-likelihood of the correct class.
  
  Note that choosing a natural loss leads to several practical advantages. In particular, applying a quadratic loss after a sigmoid activation leads to very poor gradients when the sigmoid is saturated in the wrong direction. The negative log-likelihood loss has no such problems.
  
  This issue is not specific for neural networks. Logistic regression has used the negative log likelihood loss ever since 1958.
  
  > **Comments:**
  > **username_2**: Minimize **negative** log-likelihood, or maximize log-likelihood, right? :D  
  
  ---
  
  