Answers.Body: <p>The loss function you are minimising for classification is a cross-entropy between modelled probability <span class="math-container">$p(y| f(x, \theta))$</span> and true probability <span class="math-container">$p_{\text{gt}}(y| x)$</span>:
  <span class="math-container">$$
  \text{CrossEntropy}(p, p_{\text{gt}}) = -\sum_c p_{\text{gt}}(c| x) \text{log}(p(c| f(x, \theta)))
  $$</span>
  However for observed points, you already know <span class="math-container">$p_{\text{gt}}(y| x) $</span>. It's basically 1 for class you are observing and 0 everywhere else. In other words, for the class ( lets say its c ) you observe:
  <span class="math-container">$$p_{\text{gt}}(y| x) =\delta(y = c)$$</span>
  So cross-entropy transforms into:
  <span class="math-container">$$
  \text{CrossEntropy}(p, p_{\text{gt}}) = -\sum_c \delta(y = c)\text{log}(p(y| f(x, \theta))) = -\text{log}(p(c| f(x, \theta)))
  $$</span>
  Which is exactly the formula you mention in your question ( except for  notation, in my example y is replaced by c ).</p>
  <p>As you said, the good model is the one that has a low loss function. Sine we have found, that the loss function for model, that predicts <span class="math-container">$p(y| f(x, \theta))$</span> is basically:
  <span class="math-container">$$
  \sum_c -\text{log}(p(c| f(x, \theta)))
  $$</span>
  It's easy to see, that the model, that fits the training set perfectly will have:
  <span class="math-container">\begin{align*}
  \sum_c -\text{log}(p(c| f(x, \theta))) &amp;= 0 \\ 
  \forall c,  -\text{log}(p(c| f(x, \theta))) &amp;= 0 \\ 
  \forall c,  p(c| f(x, \theta)) &amp;= 1 \\ 
  \end{align*}</span>
  Therefore, a good model (with low loss) is one that assigns a high probability to the true output <span class="math-container">$y$</span> for each corresponding input <span class="math-container">$x$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-25T01:16:04.980Z
Answers.Text: I don't see how this answers my question.
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-25T09:01:17.760Z
Answers.Text: I updated my answer to make the connection more clear. Hope it helps!
Answers.UserDisplayName: vl_knd
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-24T13:38:02.353Z
Answers.LastActivityDate: 2024-02-25T09:00:57.857Z
Answers.LastEditDate: 2024-02-25T09:00:57.857Z
Answers.UserDisplayName: vl_knd
Answers.Body: <p>Note that your context here is supervised classification and your LHS is not the usual empirical risk of the whole training set but the loss function between the true target label given an arbitrary input data and the learned model's output label given the same input data. And since you already understood the relation between log-probabilities and softmax function, it's easy to see the larger the conditional probability <span class="math-container">$p(y \mid f(\mathbf{x} ; \mathbf{\theta}))$</span> in the RHS of your equation, the smaller the loss should be, and the negative log-probability fits the bill intuitively.</p>
  <p>Of course as your reference asserted KL divergence from a later chapter formalizes the notion of loss function above along with a Dirac delta &quot;one-hot&quot; distribution <span class="math-container">$p(Y|x)=\mathbb{I}(Y = c)$</span> to represent the degenerate distribution of any above observed true target label <span class="math-container">$c$</span>, and <span class="math-container">$q(Y|x)$</span> is the above learned model's output label given the same input data. Thus minimizing loss means minimizing KL divergence, and this leads to minimizing the cross entropy of <span class="math-container">$p$</span> and <span class="math-container">$q$</span>, and <span class="math-container">$q$</span> is shown to have the exact form of negative-log probability of <span class="math-container">$c$</span> (i.e., <span class="math-container">$Y|x$</span>) on page 179.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-26T05:23:06.237Z
Answers.LastActivityDate: 2024-02-26T05:23:06.237Z
Answers.UserDisplayName: cinch
Body: <p>Chapter <strong>1.2.1.6 Maximum likelihood estimation</strong> of <a href="https://probml.github.io/pml-book/book1.html" rel="nofollow noreferrer"><em>Probabilistic Machine Learning: An Introduction</em> by Kevin P. Murphy</a> says the following:</p>
  <blockquote>
  <p>When fitting probabilistic models, it is common to use the negative log probability as our loss function:
  <span class="math-container">$$\mathcal{l}(y, f(\mathbf{x}; \mathbf{\theta})) = -\log p(y \mid f(\mathbf{x} ; \mathbf{\theta})) \tag{1.13}$$</span>
  The reasons for this are explained in Section 5.1.6.1, but intuition is that a good model (with low loss) is one that assigns a high probability to the true output <span class="math-container">$y$</span> for each corresponding input <span class="math-container">$\mathbf{x}$</span>.</p>
  </blockquote>
  <p>How does &quot;a good model (with low loss) is one that assigns a high probability to the true output <span class="math-container">$y$</span> for each corresponding input <span class="math-container">$\mathbf{x}$</span>&quot; relate to <span class="math-container">$\mathcal{l}(y, f(\mathbf{x}; \mathbf{\theta})) = -\log p(y \mid f(\mathbf{x} ; \mathbf{\theta}))$</span>?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-24T13:23:51.933Z
Comments.Text: Can you clarify whether you know what each of the symbols mean in the loss function definition? The equation and the statement about it you are questioning are strongly linked (and the intuition is solid if you understand the equation). Before attempting to write an answer, I'd like to know how you are reading/understanding the equation, so that I don'thave to write a 10-paragraph introduction to how logarithms work, if you already know
Comments.UserDisplayName: Neil Slater
Tags: <machine-learning><supervised-learning><maximum-likelihood>
Title: "a good model (with low loss) is one that assigns a high probability to the true output $y$ for each corresponding input $\mathbf{x}$"?
UserDisplayName: The Pointer
ThreadText: # "a good model (with low loss) is one that assigns a high probability to the true output $y$ for each corresponding input $\mathbf{x}$"?
  
  (Asked by: username_0 on 2024-02-24)
  
  Chapter **1.2.1.6 Maximum likelihood estimation** of [*Probabilistic Machine Learning: An Introduction* by Kevin P. Murphy](https://probml.github.io/pml-book/book1.html) says the following:
  
  > When fitting probabilistic models, it is common to use the negative log probability as our loss function:
  > $$\mathcal{l}(y, f(\mathbf{x}; \mathbf{\theta})) = -\log p(y \mid f(\mathbf{x} ; \mathbf{\theta})) \tag{1.13}$$
  > The reasons for this are explained in Section 5.1.6.1, but intuition is that a good model (with low loss) is one that assigns a high probability to the true output $y$ for each corresponding input $\mathbf{x}$.
  
  How does "a good model (with low loss) is one that assigns a high probability to the true output $y$ for each corresponding input $\mathbf{x}$" relate to $\mathcal{l}(y, f(\mathbf{x}; \mathbf{\theta})) = -\log p(y \mid f(\mathbf{x} ; \mathbf{\theta}))$?
  
  **Tags:** machine-learning, supervised-learning, maximum-likelihood
  > **Comments:**
  > **username_1**: Can you clarify whether you know what each of the symbols mean in the loss function definition? The equation and the statement about it you are questioning are strongly linked (and the intuition is solid if you understand the equation). Before attempting to write an answer, I'd like to know how you are reading/understanding the equation, so that I don'thave to write a 10-paragraph introduction to how logarithms work, if you already know  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-02-24. Score: 1
  
  The loss function you are minimising for classification is a cross-entropy between modelled probability $p(y| f(x, \theta))$ and true probability $p\_{\text{gt}}(y| x)$:
  $$
  \text{CrossEntropy}(p, p\_{\text{gt}}) = -\sum\_c p\_{\text{gt}}(c| x) \text{log}(p(c| f(x, \theta)))
  $$
  However for observed points, you already know $p\_{\text{gt}}(y| x) $. It's basically 1 for class you are observing and 0 everywhere else. In other words, for the class ( lets say its c ) you observe:
  $$p\_{\text{gt}}(y| x) =\delta(y = c)$$
  So cross-entropy transforms into:
  $$
  \text{CrossEntropy}(p, p\_{\text{gt}}) = -\sum\_c \delta(y = c)\text{log}(p(y| f(x, \theta))) = -\text{log}(p(c| f(x, \theta)))
  $$
  Which is exactly the formula you mention in your question ( except for notation, in my example y is replaced by c ).
  
  As you said, the good model is the one that has a low loss function. Sine we have found, that the loss function for model, that predicts $p(y| f(x, \theta))$ is basically:
  $$
  \sum\_c -\text{log}(p(c| f(x, \theta)))
  $$
  It's easy to see, that the model, that fits the training set perfectly will have:
  \begin{align\*}
  \sum\_c -\text{log}(p(c| f(x, \theta))) &= 0 \\
  \forall c, -\text{log}(p(c| f(x, \theta))) &= 0 \\
  \forall c, p(c| f(x, \theta)) &= 1 \\
  \end{align\*}
  Therefore, a good model (with low loss) is one that assigns a high probability to the true output $y$ for each corresponding input $x$.
  
  > **Comments:**
  > **username_0**: I don't see how this answers my question.  
  > **username_2**: I updated my answer to make the connection more clear. Hope it helps!  
  
  ---
  
  ### Answer by username_3 on 2024-02-26. Score: 0
  
  Note that your context here is supervised classification and your LHS is not the usual empirical risk of the whole training set but the loss function between the true target label given an arbitrary input data and the learned model's output label given the same input data. And since you already understood the relation between log-probabilities and softmax function, it's easy to see the larger the conditional probability $p(y \mid f(\mathbf{x} ; \mathbf{\theta}))$ in the RHS of your equation, the smaller the loss should be, and the negative log-probability fits the bill intuitively.
  
  Of course as your reference asserted KL divergence from a later chapter formalizes the notion of loss function above along with a Dirac delta "one-hot" distribution $p(Y|x)=\mathbb{I}(Y = c)$ to represent the degenerate distribution of any above observed true target label $c$, and $q(Y|x)$ is the above learned model's output label given the same input data. Thus minimizing loss means minimizing KL divergence, and this leads to minimizing the cross entropy of $p$ and $q$, and $q$ is shown to have the exact form of negative-log probability of $c$ (i.e., $Y|x$) on page 179.
  
  
  ---
  
  