Answers.Body: <p>Your intuition is correct. The restrictions you wrote down are necessary by definition.</p>
  <p>The author means that it is hard to build a machine learning model that gives probability by design ( returns <span class="math-container">$f_c$</span> directly). However we know very well how to model a function that outputs some real number. So the way to go is to create a model that gives some unnormalised log-probability ( essentially any number ) first and then transform it into <span class="math-container">$f_c$</span> by softmax function.</p>
  <p>Hope it helps.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-24T11:10:42.297Z
Answers.LastActivityDate: 2024-02-24T11:10:42.297Z
Answers.UserDisplayName: vl_knd
Body: <p>Chapter <strong>1.2.1.5 Uncertainty</strong> of <a href="https://probml.github.io/pml-book/book1.html" rel="nofollow noreferrer"><em>Probabilistic Machine Learning: An Introduction</em> by Kevin P. Murphy</a> says the following:</p>
  <blockquote>
  <p>We can capture our uncertainty using the following <strong>conditional probability distribution</strong>:
  <span class="math-container">$$p(y = c \mid \mathbf{x}; \mathbf{\theta}) = f_c(\mathbf{x}; \mathbf{\theta}) \tag{1.7}$$</span>
  where <span class="math-container">$f: \chi \to [0, 1]^C$</span> maps inputs to a probability distribution over the <span class="math-container">$C$</span> possible output labels. Since <span class="math-container">$f_c(\mathbf{x}; \mathbf{\theta})$</span> returns the probability of class label <span class="math-container">$c$</span>, we require <span class="math-container">$0 \le f_c \le 1$</span> for each <span class="math-container">$c$</span>, and <span class="math-container">$\sum_{c = 1}^C f_c = 1$</span>. To avoid this restriction, it is common to instead require the model to return unnormalized log-probabilities. We can then convert these to probabilities using the <strong>softmax function</strong>, which is defined as follows
  <span class="math-container">$$\text{softmax}(\mathbf{a}) \triangleq \left[ \dfrac{e^{a_1}}{\sum_{c^\prime = 1}^C e^{a_{c^\prime}}}, \dots, \dfrac{e^{a_C}}{\sum_{c^\prime = 1}^C e^{a_{c^\prime}}} \right] \tag{1.8}$$</span>
  This maps <span class="math-container">$\mathbb{R}^C$</span> to <span class="math-container">$[0, 1]^C$</span>, and satisfies the constraints that <span class="math-container">$0 \le \text{softmax}(\mathbf{a})_c \le 1$</span> and <span class="math-container">$\sum_{c = 1}^C \text{softmax}(\mathbf{a})_c = 1$</span>.</p>
  </blockquote>
  <p>But why would we want to &quot;avoid this restriction&quot;? <span class="math-container">$f_c$</span> is a probability, so the restrictions <span class="math-container">$0 \le f_c \le 1$</span> for each <span class="math-container">$c$</span> and <span class="math-container">$\sum_{c = 1}^C f_c = 1$</span> are necessary by definition. Also, why would we do this and then require the model to return log-probabilities only to then use the softmax function to convert it back into a probability?</p>
  
Tags: <machine-learning><supervised-learning><softmax>
Title: Since $f_c$ returns the probability of class label $c$, we require $0 \le f_c \le 1$ for each $c$, and $\sum_{c = 1}^C f_c = 1$. Why avoid this?
UserDisplayName: The Pointer
ThreadText: # Since $f_c$ returns the probability of class label $c$, we require $0 \le f_c \le 1$ for each $c$, and $\sum_{c = 1}^C f_c = 1$. Why avoid this?
  
  (Asked by: username_0 on 2024-02-24)
  
  Chapter **1.2.1.5 Uncertainty** of [*Probabilistic Machine Learning: An Introduction* by Kevin P. Murphy](https://probml.github.io/pml-book/book1.html) says the following:
  
  > We can capture our uncertainty using the following **conditional probability distribution**:
  > $$p(y = c \mid \mathbf{x}; \mathbf{\theta}) = f\_c(\mathbf{x}; \mathbf{\theta}) \tag{1.7}$$
  > where $f: \chi \to [0, 1]^C$ maps inputs to a probability distribution over the $C$ possible output labels. Since $f\_c(\mathbf{x}; \mathbf{\theta})$ returns the probability of class label $c$, we require $0 \le f\_c \le 1$ for each $c$, and $\sum\_{c = 1}^C f\_c = 1$. To avoid this restriction, it is common to instead require the model to return unnormalized log-probabilities. We can then convert these to probabilities using the **softmax function**, which is defined as follows
  > $$\text{softmax}(\mathbf{a}) \triangleq \left[ \dfrac{e^{a\_1}}{\sum\_{c^\prime = 1}^C e^{a\_{c^\prime}}}, \dots, \dfrac{e^{a\_C}}{\sum\_{c^\prime = 1}^C e^{a\_{c^\prime}}} \right] \tag{1.8}$$
  > This maps $\mathbb{R}^C$ to $[0, 1]^C$, and satisfies the constraints that $0 \le \text{softmax}(\mathbf{a})\_c \le 1$ and $\sum\_{c = 1}^C \text{softmax}(\mathbf{a})\_c = 1$.
  
  But why would we want to "avoid this restriction"? $f\_c$ is a probability, so the restrictions $0 \le f\_c \le 1$ for each $c$ and $\sum\_{c = 1}^C f\_c = 1$ are necessary by definition. Also, why would we do this and then require the model to return log-probabilities only to then use the softmax function to convert it back into a probability?
  
  **Tags:** machine-learning, supervised-learning, softmax
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-02-24. Score: 2 | Accepted answer
  
  Your intuition is correct. The restrictions you wrote down are necessary by definition.
  
  The author means that it is hard to build a machine learning model that gives probability by design ( returns $f\_c$ directly). However we know very well how to model a function that outputs some real number. So the way to go is to create a model that gives some unnormalised log-probability ( essentially any number ) first and then transform it into $f\_c$ by softmax function.
  
  Hope it helps.
  
  
  ---
  
  