Answers.Body: <blockquote>
  <p>I am specifically asking about the probability that the value is 1 (that is, how sigmoid functions specifically check for this).</p>
  </blockquote>
  <p>They don't in general. In the quoted text, there is an explicit constraint that means this can be the case:</p>
  <blockquote>
  <blockquote>
  <p>If it is desirable to predict a probability of a <strong>binary class</strong></p>
  </blockquote>
  </blockquote>
  <p>(emphasis mine). This means that the target value <span class="math-container">$y \in \{0,1\}$</span>. Which in turn means that training data labels will all be <span class="math-container">$0$</span> or <span class="math-container">$1$</span>.</p>
  <p>At this point, if you pass in some calculated weighted sum of features to a sigmoid function, it will output a real value between <span class="math-container">$0.0$</span> and <span class="math-container">$1.0$</span> that can be interpreted as a probability. But without more going on, it is not linked in any way to the probability of the label being <span class="math-container">$1$</span>.</p>
  <p>To complete things, there is also an <em>implicit</em> constraint (not referred in your quoted text) that you will use a training approach that will cause this weighted sum of features to drive the sigmoid to represent a meaningful probability based on the source data. The most important part of this is to use a cost function which is minimised when <span class="math-container">$\hat{y} = \mathbb{P} \{y = 1 \mid \mathbf{x}\}$</span> where <span class="math-container">$\mathbf{x}$</span> is the input features associated with a label. The most usual cost function here would be based on binary cross-entropy loss:</p>
  <p><span class="math-container">$$C = -\frac{1}{N}\sum_{i=1}^{N}y_i\text{log}(\hat{y}_i) + (1-y_i)\text{log}(1 -\hat{y}_i)$$</span></p>
  <p>You can show using calculus that this term is minimised when <span class="math-container">$\hat{y}$</span> is directly related to the frequency of <span class="math-container">$y = 1$</span> in the training data. Adding features <span class="math-container">$\mathbf{x}$</span> makes this more complex, but as a first pass to understanding what this is doing, you can simply use a list of <span class="math-container">$N$</span> times <span class="math-container">$0$</span> or <span class="math-container">$1$</span> outputs with no input data, and show that the sum above is minimised by a fixed <span class="math-container">$\hat{y}$</span> equal to the proportion of <span class="math-container">$y = 1$</span> – the rationale is that you would use a fixed <span class="math-container">$\hat{y}$</span> as your guess for each value if you had no input data about any example to go on.</p>
  <p>One important detail of the related maths is that the activation function does not affect the target of the convergence. However, it may still affect the speed and stability of the convergence. The combination of binary cross-entropy loss and sigmoid activation is popular because the gradient calculation is simple and relatively stable.</p>
  <hr />
  <p>Here is the maths worked through for a population of <span class="math-container">$N$</span> examples <span class="math-container">$y_i$</span> with no feature data (so no <span class="math-container">$\mathbf{x}_i$</span>), <span class="math-container">$M$</span> of which have <span class="math-container">$y_i = 1$</span> and <span class="math-container">$N - M$</span> of which have <span class="math-container">$y_i=0$</span>. We will guess a fixed value of <span class="math-container">$\hat{y}$</span> so that it minimises the mean cross-entropy loss:</p>
  <p><span class="math-container">$$\text{argmin}_{\hat{y}} C  = \text{argmin}_{\hat{y}}-\frac{1}{N}\sum_{i=1}^{N}y_i\text{log}(\hat{y}) + (1-y_i)\text{log}(1 -\hat{y})$$</span></p>
  <p>The factor of <span class="math-container">$\frac{1}{N}$</span> can be removed as it doesn't affect what value of <span class="math-container">$\hat{y}$</span> causes the minimum. In addition, we have <span class="math-container">$M$</span> values where <span class="math-container">$y = 1$</span> and <span class="math-container">$N - M$</span> values where <span class="math-container">$y = 0$</span>, so we can simplify the sum.</p>
  <p><span class="math-container">$$= \text{argmin}_{\hat{y}}-M\text{log}(\hat{y}) - (N-M)\text{log}(1 -\hat{y})$$</span></p>
  <p>To find the stationary point, take the derivative and set equal to zero (I reversed signs here since <span class="math-container">$-0 = 0$</span>).</p>
  <p><span class="math-container">$$\frac{d}{d\hat{y}}[ M\text{log}(\hat{y}) + (N-M)\text{log}(1 -\hat{y})] = 0$$</span></p>
  <p><span class="math-container">$$\frac{M}{\hat{y}} - \frac{N-M}{1-\hat{y}} = 0$$</span></p>
  <p><span class="math-container">$$(1-\hat{y})M - \hat{y}(N-M) = 0$$</span></p>
  <p><span class="math-container">$$M = \hat{y}N$$</span></p>
  <p><span class="math-container">$$\hat{y} = \frac{M}{N}$$</span></p>
  <p>This of course is the same as the probability that a randomly selected <span class="math-container">$i \in (1,N)$</span> will have <span class="math-container">$y_i = 1$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-12T01:29:09.623Z
Answers.Text: Wow, such an amazing, illuminating answer! Thank you very for taking the time to write this.
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-11T13:01:11.900Z
Answers.LastActivityDate: 2021-06-13T12:26:41.900Z
Answers.LastEditDate: 2021-06-13T12:26:41.900Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am currently studying the textbook <em>Neural Networks and Deep Learning</em> by Charu C. Aggarwal. Chapter <strong>1.2.1.3 Choice of Activation and Loss Functions</strong> says the following:</p>
  <blockquote>
  <p>The choice of activation function is a critical part of neural network design. In the case of the perceptron, the choice of the sign activation function is motivated by the fact that a binary class label needs to be predicted. However, it is possible to have other types of situations where different target variables may be predicted. For example, if the target variable to be predicted is real, then it makes sense to use the identity activation function, and the resulting algorithm is the same as least-squares regression. If it is desirable to predict a probability of a binary class, it makes sense to use a sigmoid function for activating the output node, so that the prediction <span class="math-container">$\hat{y}$</span> indicates the probability that the observed value, <span class="math-container">$y$</span>, of the dependent variable is <span class="math-container">$1$</span>.</p>
  </blockquote>
  <p>I've read about <a href="https://en.wikipedia.org/wiki/Sigmoid_function" rel="nofollow noreferrer">sigmoid functions</a>, but it isn't clear to me how they make it so that the prediction <span class="math-container">$\hat{y}$</span> indicates the probability that the observed value, <span class="math-container">$y$</span>, of the dependent variable is <span class="math-container">$1$</span>. So how do sigmoid functions make it so that the prediction <span class="math-container">$\hat{y}$</span> indicates the probability that the observed value, <span class="math-container">$y$</span>, of the dependent variable is <span class="math-container">$1$</span>?</p>
  <p>EDIT: I am specifically asking about the probability that the value is <span class="math-container">$1$</span> (that is, how sigmoid functions specifically check for this).</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-11T11:11:52.160Z
Comments.Text: Not clear whether you are asking about the target value $1$, or how sigmoid can represent probabilities at all, or how using sigmoid will result in a meaningful probability measure in practice.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-11T11:19:23.363Z
Comments.Text: Hi @NeilSlater , I am specifically asking about the probability that the value is $1$ (that is, how sigmoid functions specifically check for this). I have edited my question for clarity.
Comments.UserDisplayName: The Pointer
Tags: <neural-networks><activation-functions><perceptron><sigmoid>
Title: How do sigmoid functions make it so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, is $1$?
UserDisplayName: The Pointer
ThreadText: # How do sigmoid functions make it so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, is $1$?
  
  (Asked by: username_0 on 2021-06-11)
  
  I am currently studying the textbook *Neural Networks and Deep Learning* by Charu C. Aggarwal. Chapter **1.2.1.3 Choice of Activation and Loss Functions** says the following:
  
  > The choice of activation function is a critical part of neural network design. In the case of the perceptron, the choice of the sign activation function is motivated by the fact that a binary class label needs to be predicted. However, it is possible to have other types of situations where different target variables may be predicted. For example, if the target variable to be predicted is real, then it makes sense to use the identity activation function, and the resulting algorithm is the same as least-squares regression. If it is desirable to predict a probability of a binary class, it makes sense to use a sigmoid function for activating the output node, so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, of the dependent variable is $1$.
  
  I've read about [sigmoid functions](https://en.wikipedia.org/wiki/Sigmoid_function), but it isn't clear to me how they make it so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, of the dependent variable is $1$. So how do sigmoid functions make it so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, of the dependent variable is $1$?
  
  EDIT: I am specifically asking about the probability that the value is $1$ (that is, how sigmoid functions specifically check for this).
  
  **Tags:** neural-networks, activation-functions, perceptron, sigmoid
  
  > **Comments:**
  > **username_1**: Not clear whether you are asking about the target value $1$, or how sigmoid can represent probabilities at all, or how using sigmoid will result in a meaningful probability measure in practice.  
  > **username_0**: Hi @username_1 , I am specifically asking about the probability that the value is $1$ (that is, how sigmoid functions specifically check for this). I have edited my question for clarity.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-06-11. Score: 3 | Accepted answer
  
  > I am specifically asking about the probability that the value is 1 (that is, how sigmoid functions specifically check for this).
  
  They don't in general. In the quoted text, there is an explicit constraint that means this can be the case:
  
  > > If it is desirable to predict a probability of a **binary class**
  
  (emphasis mine). This means that the target value $y \in \{0,1\}$. Which in turn means that training data labels will all be $0$ or $1$.
  
  At this point, if you pass in some calculated weighted sum of features to a sigmoid function, it will output a real value between $0.0$ and $1.0$ that can be interpreted as a probability. But without more going on, it is not linked in any way to the probability of the label being $1$.
  
  To complete things, there is also an *implicit* constraint (not referred in your quoted text) that you will use a training approach that will cause this weighted sum of features to drive the sigmoid to represent a meaningful probability based on the source data. The most important part of this is to use a cost function which is minimised when $\hat{y} = \mathbb{P} \{y = 1 \mid \mathbf{x}\}$ where $\mathbf{x}$ is the input features associated with a label. The most usual cost function here would be based on binary cross-entropy loss:
  
  $$C = -\frac{1}{N}\sum\_{i=1}^{N}y\_i\text{log}(\hat{y}\_i) + (1-y\_i)\text{log}(1 -\hat{y}\_i)$$
  
  You can show using calculus that this term is minimised when $\hat{y}$ is directly related to the frequency of $y = 1$ in the training data. Adding features $\mathbf{x}$ makes this more complex, but as a first pass to understanding what this is doing, you can simply use a list of $N$ times $0$ or $1$ outputs with no input data, and show that the sum above is minimised by a fixed $\hat{y}$ equal to the proportion of $y = 1$ – the rationale is that you would use a fixed $\hat{y}$ as your guess for each value if you had no input data about any example to go on.
  
  One important detail of the related maths is that the activation function does not affect the target of the convergence. However, it may still affect the speed and stability of the convergence. The combination of binary cross-entropy loss and sigmoid activation is popular because the gradient calculation is simple and relatively stable.
  
  ---
  
  Here is the maths worked through for a population of $N$ examples $y\_i$ with no feature data (so no $\mathbf{x}\_i$), $M$ of which have $y\_i = 1$ and $N - M$ of which have $y\_i=0$. We will guess a fixed value of $\hat{y}$ so that it minimises the mean cross-entropy loss:
  
  $$\text{argmin}\_{\hat{y}} C = \text{argmin}\_{\hat{y}}-\frac{1}{N}\sum\_{i=1}^{N}y\_i\text{log}(\hat{y}) + (1-y\_i)\text{log}(1 -\hat{y})$$
  
  The factor of $\frac{1}{N}$ can be removed as it doesn't affect what value of $\hat{y}$ causes the minimum. In addition, we have $M$ values where $y = 1$ and $N - M$ values where $y = 0$, so we can simplify the sum.
  
  $$= \text{argmin}\_{\hat{y}}-M\text{log}(\hat{y}) - (N-M)\text{log}(1 -\hat{y})$$
  
  To find the stationary point, take the derivative and set equal to zero (I reversed signs here since $-0 = 0$).
  
  $$\frac{d}{d\hat{y}}[ M\text{log}(\hat{y}) + (N-M)\text{log}(1 -\hat{y})] = 0$$
  
  $$\frac{M}{\hat{y}} - \frac{N-M}{1-\hat{y}} = 0$$
  
  $$(1-\hat{y})M - \hat{y}(N-M) = 0$$
  
  $$M = \hat{y}N$$
  
  $$\hat{y} = \frac{M}{N}$$
  
  This of course is the same as the probability that a randomly selected $i \in (1,N)$ will have $y\_i = 1$.
  
  > **Comments:**
  > **username_0**: Wow, such an amazing, illuminating answer! Thank you very for taking the time to write this.  
  
  ---
  
  