Answers.Body: <p>Loss function is a function used to measure the loss. It is not used in any component of a neuron. It is used in updating the weights of the neuron i.e., in order to train the neuron.</p>
  <p>The contribution of a loss function is in the updation of <span class="math-container">$\bar{W}$</span>.</p>
  <p>For a given <span class="math-container">$\bar{X}$</span> and <span class="math-container">$\bar{W}$</span>, the neuron gives a post-action value <span class="math-container">$h$</span>. But the desired output may not be exactly <span class="math-container">$h$</span>. The distance measure between the desired value (say <span class="math-container">$h^\prime$</span>) and post-activation value <span class="math-container">$h$</span> is called the loss of the perceptron for <span class="math-container">$\bar{X}$</span>.</p>
  <p>We want to decrease the loss of the perception i.e., <span class="math-container">$\left\vert h - h^\prime \right\vert$</span>. The only task we can do is changing <span class="math-container">$\bar{W}$</span>, the parameters of the perceptron model given in the question. In order to update the weights, we have to calculate the derivative of loss function wrt the weights of the model and then update weights using some updation rule.</p>
  <p>Thus, loss function will come into play during the training phase of the neuron.</p>
  <p>To be concise: currently you are dealing with the neuron and its components. Loss functions for training the neuron will come later.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-16T05:25:33.853Z
Answers.LastActivityDate: 2021-06-16T05:53:21.970Z
Answers.LastEditDate: 2021-06-16T05:53:21.970Z
Answers.UserDisplayName: hanugm
Answers.Body: <p>The loss function is simply a way to measure how wrong a neural network is, it doesn't affect the output of the neuron.</p>
  <p>Say we have a neural network with 3 output neurons that attempts to classify images of cats, dogs, and humans. The output it gives is the confidence of the neural network's classification. For example if the output is [0, 0.2, 0.8] (0 being the output of the 1st neuron, 0.2 of the 2nd, and 0.8 of the 3rd), this means that the neural network thinks that the image has 0% probability of being a cat, 20% of being a dog and 80% of being a human.</p>
  <p>Imagine that the image that was shown to the network is a human, we can say that the target values are [0, 0, 1] because we want it to output that the image is a human with 100% confidence. Now we must measure how wrong the prediction actually was using a loss function. There are many loss functions, but for simplicity I'll use the squared error. In this case the loss will be equal to (1-0.8)^2=0.04 (expected value - output)^2.</p>
  <p>The closer the output is to 1, the result inside of the brackets will be closer to 0, so the loss will be smaller. The objective is to minimize this loss function. For example, if the output was 1 instead of 0.8, the network's loss will be (1-1)^2 = 0. If the output was 0.2 instead, the loss would be (1-0.2)^2 = 0.64, which is larger than the two previous, as it is 'more wrong'.</p>
  <p>To train the network we use this instead of accuracy for the following reason. With both these outputs [0, 0.1, 0.9], [0.2, 0.3, 0.5] the network predicts 'human', the largest value, but in the first case it is 90% sure whereas in the second it is only 50% sure. We can say that the first network is better, but if we only used accuracy, as both predict the same, they would appear to be just as good.</p>
  <p>The same happens when they make a mistake. If the expected values are [0, 1, 0] and one model predicts [0.5, 0.4, 0.1] and the other predicts [0.9, 0, 0.1], they both got it wrong, but the first one was <em>less</em> wrong. The first loss would be (1-0.4)^2 = 0.36 and the second would be (1-0)^2 = 1, which is much higher</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-16T10:45:57.577Z
Answers.Text: @The Pointer, if I were you I would check Sentdex's Neural Networks From Scratch series. Here is the link: https://www.youtube.com/watch?v=Wo5dMEP_BbI&list=PLQVvvaa0QuDcjD5BAw2DxE6OF2tius3V3
Answers.UserDisplayName: Unnamed
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-16T10:43:30.437Z
Answers.LastActivityDate: 2021-06-16T10:43:30.437Z
Answers.UserDisplayName: Unnamed
Answers.Body: <p>Assume we have a binary classification problem, which we want to solve with a simple single-layer perceptron. For a 2d space, a perceptron will have 2 inputs <span class="math-container">$x_1$</span> and <span class="math-container">$x_2$</span>, and a bias denoted <span class="math-container">$x_0$</span> which is always <span class="math-container">$x_0=1$</span>. It also has corresponding learnable weights <span class="math-container">$w_0$</span>, <span class="math-container">$w_1$</span> and <span class="math-container">$w_2$</span>.</p>
  <p>This can be vectorized:</p>
  <p><span class="math-container">$$
  \overline{x} = \begin{bmatrix} 1 \\ x_1 \\ x_2 \end{bmatrix}; \
  \overline{w} = \begin{bmatrix} w_0 \\ w_1 \\ w_2 \end{bmatrix}
  $$</span></p>
  <p><a href="https://i.sstatic.net/6e8Ilm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6e8Ilm.png" alt="enter image description here" /></a></p>
  <p>Then the pre-activation function <span class="math-container">$a = \overline{w}\cdot\overline{x}$</span> nothing more than a linear equation and can be unfolded to <span class="math-container">$a = w_0 + w_1x_1+w_2x_2$</span>.</p>
  <p>In the simplest case <span class="math-container">$h$</span> can be defined as follows:</p>
  <p><span class="math-container">$$h  = \begin{cases}
        \text{class 1} &amp; \text{if $a&gt;0$}\\
        \text{class 2} &amp; \text{otherwise}
      \end{cases}
  $$</span></p>
  <p>or it can also be defined as <span class="math-container">$h=\text{sign}(\overline{w}\cdot\overline{x})$</span>, which means that samples above the line belong to class 1.</p>
  <p><a href="https://i.sstatic.net/HJZUq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HJZUq.png" alt="enter image description here" /></a></p>
  <p>Since our <span class="math-container">$h$</span> is not continuous and not differentiable, we cannot apply gradient descent optimization. Instead, the weights can be optimized iteratively:</p>
  <blockquote>
  <p><a href="https://i.sstatic.net/eSu0r.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/eSu0r.png" alt="enter image description here" /></a></p>
  </blockquote>
  <p>for the details, you can refer to <a href="https://en.wikipedia.org/wiki/Perceptron#Learning_algorithm" rel="nofollow noreferrer">Wikipedia</a></p>
  <p>So <strong>answering your question</strong>, for this particular case, there is no loss function. Optimization of a loss function usually involves <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent" rel="nofollow noreferrer">SGD</a> which requires the function to be continuous and differentiable. For instance, if <span class="math-container">$h$</span> is defined as a sigmoid function, you can use binary cross-entropy as a loss function.</p>
  <p>There are also other optimization approaches, such as genetic algorithm, expectation-maximization and hill-climbing that do not require a loss function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-16T12:01:54.980Z
Answers.LastActivityDate: 2021-06-16T13:09:44.183Z
Answers.LastEditDate: 2021-06-16T13:09:44.183Z
Answers.UserDisplayName: Aray Karjauv
Body: <p>I am currently studying the textbook <em>Neural Networks and Deep Learning</em> by Charu C. Aggarwal. Chapter <strong>1.2.1.3 Choice of Activation and Loss Functions</strong> presents the following figure:</p>
  <blockquote>
  <p><a href="https://i.sstatic.net/wKwkG.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wKwkG.png" alt="enter image description here" /></a></p>
  </blockquote>
  <p><span class="math-container">$\overline{X}$</span> is the features, <span class="math-container">$\overline{W}$</span> is the weights, and <span class="math-container">$\phi$</span> is the activation function.</p>
  <p>So this is a <a href="https://en.wikipedia.org/wiki/Perceptron" rel="nofollow noreferrer">perceptron</a> (which is a form of <a href="https://en.wikipedia.org/wiki/Artificial_neuron" rel="nofollow noreferrer">artificial neuron</a>).</p>
  <p>But where does the so-called 'loss' / 'loss function' fit into this? This is something that I've been unable to reconcile.</p>
  <hr />
  <h2>EDIT</h2>
  <p>The way the loss function was introduced in the textbook seemed to imply that it was part of the architecture of the perceptron / artificial neuron, but, according to hanugm's answer, it is external and instead used to <em>update the weights of the neuron</em>. So it seems that I misunderstood what was written in the textbook.</p>
  <p>In my question above, I pretty much assumed that the loss function was part of the architecture of the perceptron / artificial neuron, and then asked how it fit into the architecture, since I couldn't see any indication of it in the figure.</p>
  <p>Is the loss / loss function part of the architecture of a perceptron / artificial neuron? I cannot see any indication of a loss / loss function in figure 1.7, so I'm confused about this. If not, then how does the loss / loss function relate to the architecture of the perceptron / artificial neuron?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-16T01:04:35.260Z
Comments.Text: there is no loss function on this figure. a loss function is a measure of an error between a predicted value and the ground truth. it can be a mean squared error (for regression), or cross-entropy for a classification problem. take a look at this [list](https://keras.io/api/losses/). for a simple linear perceptron, the problem becomes a linear equation: $y=b+ax$, e.g. what is above the line ($y>0$) is class one, otherwise class two. in this case, you just need to fit a line between samples of 2 classes
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-16T08:40:30.080Z
Comments.Text: It's not clear to me what the question is here. Are you asking what is the loss function used to train the perceptron?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-16T08:44:17.883Z
Comments.Text: @nbro The way the loss function was introduced in the textbook seemed to imply that it was part of the architecture of the perceptron / artificial neuron, but, according to hanugm's answer, it is external and instead used to *update the weights of the neuron*. So it seems that I misunderstood what was written in the textbook.
Comments.UserDisplayName: The Pointer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-16T09:44:43.097Z
Comments.Text: So, is your question: _is the loss function of the perceptron somehow part of its architecture?_ If that was the question, it may be a good idea to edit your post to clarify that.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-16T09:46:33.363Z
Comments.Text: @nbro I pretty much assumed that the loss function was part of the architecture of the perceptron / artificial neuron, and then asked how it fit into the architecture, since I couldn't see any indication of it in the figure. I'll edit for clarity.
Comments.UserDisplayName: The Pointer
Tags: <objective-functions><activation-functions><artificial-neuron><perceptron><loss>
Title: Where does the so-called 'loss' / 'loss function' fit into the idea of a perceptron / artificial neuron (as presented in the figure)?
UserDisplayName: The Pointer
ThreadText: # Where does the so-called 'loss' / 'loss function' fit into the idea of a perceptron / artificial neuron (as presented in the figure)?
  
  (Asked by: username_0 on 2021-06-13)
  
  I am currently studying the textbook *Neural Networks and Deep Learning* by Charu C. Aggarwal. Chapter **1.2.1.3 Choice of Activation and Loss Functions** presents the following figure:
  
  > [![enter image description here](https://i.sstatic.net/wKwkG.png)](https://i.sstatic.net/wKwkG.png)
  
  $\overline{X}$ is the features, $\overline{W}$ is the weights, and $\phi$ is the activation function.
  
  So this is a [perceptron](https://en.wikipedia.org/wiki/Perceptron) (which is a form of [artificial neuron](https://en.wikipedia.org/wiki/Artificial_neuron)).
  
  But where does the so-called 'loss' / 'loss function' fit into this? This is something that I've been unable to reconcile.
  
  ---
  
  ## EDIT
  
  The way the loss function was introduced in the textbook seemed to imply that it was part of the architecture of the perceptron / artificial neuron, but, according to username_3's answer, it is external and instead used to *update the weights of the neuron*. So it seems that I misunderstood what was written in the textbook.
  
  In my question above, I pretty much assumed that the loss function was part of the architecture of the perceptron / artificial neuron, and then asked how it fit into the architecture, since I couldn't see any indication of it in the figure.
  
  Is the loss / loss function part of the architecture of a perceptron / artificial neuron? I cannot see any indication of a loss / loss function in figure 1.7, so I'm confused about this. If not, then how does the loss / loss function relate to the architecture of the perceptron / artificial neuron?
  
  **Tags:** objective-functions, activation-functions, artificial-neuron, perceptron, loss
  
  > **Comments:**
  > **username_1**: there is no loss function on this figure. a loss function is a measure of an error between a predicted value and the ground truth. it can be a mean squared error (for regression), or cross-entropy for a classification problem. take a look at this [list](https://keras.io/api/losses/). for a simple linear perceptron, the problem becomes a linear equation: $y=b+ax$, e.g. what is above the line ($y>0$) is class one, otherwise class two. in this case, you just need to fit a line between samples of 2 classes  
  > **username_2**: It's not clear to me what the question is here. Are you asking what is the loss function used to train the perceptron?  
  > **username_0**: @username_2 The way the loss function was introduced in the textbook seemed to imply that it was part of the architecture of the perceptron / artificial neuron, but, according to username_3's answer, it is external and instead used to *update the weights of the neuron*. So it seems that I misunderstood what was written in the textbook.  
  > **username_2**: So, is your question: _is the loss function of the perceptron somehow part of its architecture?_ If that was the question, it may be a good idea to edit your post to clarify that.  
  > **username_0**: @username_2 I pretty much assumed that the loss function was part of the architecture of the perceptron / artificial neuron, and then asked how it fit into the architecture, since I couldn't see any indication of it in the figure. I'll edit for clarity.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2021-06-16. Score: 1 | Accepted answer
  
  Loss function is a function used to measure the loss. It is not used in any component of a neuron. It is used in updating the weights of the neuron i.e., in order to train the neuron.
  
  The contribution of a loss function is in the updation of $\bar{W}$.
  
  For a given $\bar{X}$ and $\bar{W}$, the neuron gives a post-action value $h$. But the desired output may not be exactly $h$. The distance measure between the desired value (say $h^\prime$) and post-activation value $h$ is called the loss of the perceptron for $\bar{X}$.
  
  We want to decrease the loss of the perception i.e., $\left\vert h - h^\prime \right\vert$. The only task we can do is changing $\bar{W}$, the parameters of the perceptron model given in the question. In order to update the weights, we have to calculate the derivative of loss function wrt the weights of the model and then update weights using some updation rule.
  
  Thus, loss function will come into play during the training phase of the neuron.
  
  To be concise: currently you are dealing with the neuron and its components. Loss functions for training the neuron will come later.
  
  
  ---
  
  ### Answer by username_4 on 2021-06-16. Score: 1
  
  The loss function is simply a way to measure how wrong a neural network is, it doesn't affect the output of the neuron.
  
  Say we have a neural network with 3 output neurons that attempts to classify images of cats, dogs, and humans. The output it gives is the confidence of the neural network's classification. For example if the output is [0, 0.2, 0.8] (0 being the output of the 1st neuron, 0.2 of the 2nd, and 0.8 of the 3rd), this means that the neural network thinks that the image has 0% probability of being a cat, 20% of being a dog and 80% of being a human.
  
  Imagine that the image that was shown to the network is a human, we can say that the target values are [0, 0, 1] because we want it to output that the image is a human with 100% confidence. Now we must measure how wrong the prediction actually was using a loss function. There are many loss functions, but for simplicity I'll use the squared error. In this case the loss will be equal to (1-0.8)^2=0.04 (expected value - output)^2.
  
  The closer the output is to 1, the result inside of the brackets will be closer to 0, so the loss will be smaller. The objective is to minimize this loss function. For example, if the output was 1 instead of 0.8, the network's loss will be (1-1)^2 = 0. If the output was 0.2 instead, the loss would be (1-0.2)^2 = 0.64, which is larger than the two previous, as it is 'more wrong'.
  
  To train the network we use this instead of accuracy for the following reason. With both these outputs [0, 0.1, 0.9], [0.2, 0.3, 0.5] the network predicts 'human', the largest value, but in the first case it is 90% sure whereas in the second it is only 50% sure. We can say that the first network is better, but if we only used accuracy, as both predict the same, they would appear to be just as good.
  
  The same happens when they make a mistake. If the expected values are [0, 1, 0] and one model predicts [0.5, 0.4, 0.1] and the other predicts [0.9, 0, 0.1], they both got it wrong, but the first one was *less* wrong. The first loss would be (1-0.4)^2 = 0.36 and the second would be (1-0)^2 = 1, which is much higher
  
  > **Comments:**
  > **username_4**: @username_0, if I were you I would check Sentdex's Neural Networks From Scratch series. Here is the link: https://www.youtube.com/watch?v=Wo5dMEP_BbI&list=PLQVvvaa0QuDcjD5BAw2DxE6OF2tius3V3  
  
  ---
  
  ### Answer by username_1 on 2021-06-16. Score: 1
  
  Assume we have a binary classification problem, which we want to solve with a simple single-layer perceptron. For a 2d space, a perceptron will have 2 inputs $x\_1$ and $x\_2$, and a bias denoted $x\_0$ which is always $x\_0=1$. It also has corresponding learnable weights $w\_0$, $w\_1$ and $w\_2$.
  
  This can be vectorized:
  
  $$
  \overline{x} = \begin{bmatrix} 1 \\ x\_1 \\ x\_2 \end{bmatrix}; \
  \overline{w} = \begin{bmatrix} w\_0 \\ w\_1 \\ w\_2 \end{bmatrix}
  $$
  
  [![enter image description here](https://i.sstatic.net/6e8Ilm.png)](https://i.sstatic.net/6e8Ilm.png)
  
  Then the pre-activation function $a = \overline{w}\cdot\overline{x}$ nothing more than a linear equation and can be unfolded to $a = w\_0 + w\_1x\_1+w\_2x\_2$.
  
  In the simplest case $h$ can be defined as follows:
  
  $$h = \begin{cases}
  \text{class 1} & \text{if $a>0$}\\
  \text{class 2} & \text{otherwise}
  \end{cases}
  $$
  
  or it can also be defined as $h=\text{sign}(\overline{w}\cdot\overline{x})$, which means that samples above the line belong to class 1.
  
  [![enter image description here](https://i.sstatic.net/HJZUq.png)](https://i.sstatic.net/HJZUq.png)
  
  Since our $h$ is not continuous and not differentiable, we cannot apply gradient descent optimization. Instead, the weights can be optimized iteratively:
  
  > [![enter image description here](https://i.sstatic.net/eSu0r.png)](https://i.sstatic.net/eSu0r.png)
  
  for the details, you can refer to [Wikipedia](https://en.wikipedia.org/wiki/Perceptron#Learning_algorithm)
  
  So **answering your question**, for this particular case, there is no loss function. Optimization of a loss function usually involves [SGD](https://en.wikipedia.org/wiki/Stochastic_gradient_descent) which requires the function to be continuous and differentiable. For instance, if $h$ is defined as a sigmoid function, you can use binary cross-entropy as a loss function.
  
  There are also other optimization approaches, such as genetic algorithm, expectation-maximization and hill-climbing that do not require a loss function.
  
  
  ---
  
  