Answers.Body: <p>From Bayesian statistics maximizing the data likelihood given a hypothesis of a specific parametric statistical model to be regressed along with a specific noise as source randomness is equivalent to <em>minimizing</em> the negative log-likelihood (NLL) of a <em>specific</em> form, for instance, from your previous question you've already known for any linear regression model with a Gaussian distributed additive noise the maximum likelihood approach demands the form of squared error loss. Thus the maximum likelihood approach is a generic way for <em>systematically</em> constructing a loss function based on a parametric model of the observed data, <em>ad hoc</em> loss functions can be avoided in theory for any assumed generic parametric model with any generically distributed noise.</p>
  <p>If the error is non-symmetric, then obviously you cannot assume it following the usual Gaussian or Laplace distributions, perhaps you could assume it following the <a href="https://en.wikipedia.org/wiki/Skew_normal_distribution" rel="nofollow noreferrer">skew normal distribution</a>, <a href="https://en.wikipedia.org/wiki/Beta_distribution" rel="nofollow noreferrer">beta distribution</a>, etc. Of course for a linear regression model with a non-symmetric noise the NLL does not typically have a closed-form analytic solution, so to find the parameters minimizing the NLL would usually involve iterative numerical optimization methods such as gradient descents.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-15T08:24:12.223Z
Answers.LastActivityDate: 2023-11-15T22:01:18.600Z
Answers.LastEditDate: 2023-11-15T22:01:18.600Z
Answers.UserDisplayName: cinch
Body: <p>When the probability of observing a large positive error is larger than the probability
  of observing a large negative error in binary classification, how can  this be modelled by a skewed noise distribution using the maximum likelihood approach?</p>
  <p>The content is as follows:</p>
  <p>From page 103 of: <a href="http://smlbook.org/book/sml-book-draft-latest.pdf" rel="nofollow noreferrer">http://smlbook.org/book/sml-book-draft-latest.pdf</a></p>
  <p>&quot;Using the maximum likelihood approach, other assumptions about the noise or
  insights into its distribution can be incorporated in a similar way in the regression
  model (5.1). For instance, if we believe that the error is non-symmetric, in the sense
  that the probability of observing a large positive error is larger than the probability
  of observing a large negative error, then this can be modelled by a skewed noise
  distribution. Using the negative log-likelihood loss is then a systematic way of
  incorporating this skewness into the training objective.&quot;</p>
  
Tags: <machine-learning><statistical-ai>
Title: How can we construct a skewed noise distribution using the maximum likelihood approach?
UserDisplayName: DSPinfinity
ThreadText: # How can we construct a skewed noise distribution using the maximum likelihood approach?
  
  (Asked by: username_0 on 2023-11-14)
  
  When the probability of observing a large positive error is larger than the probability
  of observing a large negative error in binary classification, how can this be modelled by a skewed noise distribution using the maximum likelihood approach?
  
  The content is as follows:
  
  From page 103 of: <http://smlbook.org/book/sml-book-draft-latest.pdf>
  
  "Using the maximum likelihood approach, other assumptions about the noise or
  insights into its distribution can be incorporated in a similar way in the regression
  model (5.1). For instance, if we believe that the error is non-symmetric, in the sense
  that the probability of observing a large positive error is larger than the probability
  of observing a large negative error, then this can be modelled by a skewed noise
  distribution. Using the negative log-likelihood loss is then a systematic way of
  incorporating this skewness into the training objective."
  
  **Tags:** machine-learning, statistical-ai
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-15. Score: 0
  
  From Bayesian statistics maximizing the data likelihood given a hypothesis of a specific parametric statistical model to be regressed along with a specific noise as source randomness is equivalent to *minimizing* the negative log-likelihood (NLL) of a *specific* form, for instance, from your previous question you've already known for any linear regression model with a Gaussian distributed additive noise the maximum likelihood approach demands the form of squared error loss. Thus the maximum likelihood approach is a generic way for *systematically* constructing a loss function based on a parametric model of the observed data, *ad hoc* loss functions can be avoided in theory for any assumed generic parametric model with any generically distributed noise.
  
  If the error is non-symmetric, then obviously you cannot assume it following the usual Gaussian or Laplace distributions, perhaps you could assume it following the [skew normal distribution](https://en.wikipedia.org/wiki/Skew_normal_distribution), [beta distribution](https://en.wikipedia.org/wiki/Beta_distribution), etc. Of course for a linear regression model with a non-symmetric noise the NLL does not typically have a closed-form analytic solution, so to find the parameters minimizing the NLL would usually involve iterative numerical optimization methods such as gradient descents.
  
  
  ---
  
  