Answers.Body: <p>Figure 1.6 is depicting that the hinge-loss used in SVMs (Eqn 1.9):</p>
  <p><span class="math-container">$$L_i^{svm} = \text{max}\{1 - y_i (\overline{W} \cdot \overline{X}_i, 0\} \tag{1.9}$$</span></p>
  <p>is a shifted version (+1 to the right) of the loss function (Eqn 1.8) used to optimize the perceptron criterion (see footnote (1)):</p>
  <p><span class="math-container">$$L_i = \text{max}\{-y_i (\overline{W} \cdot \overline{X}_i), 0\} \tag{1.8}$$</span></p>
  <p>Using Figure 1.6 and the accompanying discussion in Section 1.2.1.2, the author (Charu C. Aggarwal) is trying to show that if you make the implicit training process of the perceptron explicit (with an objective function, in his case, a loss function defined by Eqn 1.8), then one realises (in his words):</p>
  <blockquote>
  <p>that the perceptron is fundamentally not very different from well-known machine learning algorithms like the support vector machine in spite of its different origins.</p>
  </blockquote>
  <h3>Footnotes</h3>
  <p>(1) The <em>perceptron criterion</em> was introduced in Section 3.5.1 of <a href="https://www.academia.edu/download/55593223/_Christopher_M._Bishop__Neural_Networks_for_Patterb-ok.org.pdf" rel="nofollow noreferrer">Neural Networks for Pattern Recognition</a> by Christopher Bishop as a <em>continuous</em>, piecewise linear error function given by:</p>
  <p><span class="math-container">$$E^{\text{perc}}(\bf{w}) = - \sum_{\phi^n \in \mathcal{m}} w^T (\phi^n t^n)$$</span></p>
  <p>Bishop suggested that this objective function is minimised when training a perceptron. Aggarwal references this work as an example of how the perceptron training process implicitly minimises this error function, using it as a segue into his own derivation of Eqn 1.8 before encouraging the reader to check that the gradient of Eqn 1.8 leads to the perceptron update (i.e. <span class="math-container">$\overline{W} \Leftarrow \overline{W} - \alpha \nabla_{W} L_i$</span>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-05T14:16:15.457Z
Answers.LastActivityDate: 2022-11-05T14:16:15.457Z
Answers.UserDisplayName: Kyle F. Hartzenberg
Answers.Body: <p>I can't fully explain the part because I forgot what it talks about.</p>
  <p>However, regarding the hinge loss, it is basically allowing your SVM to tolerate misclassifications without increasing the cost function.</p>
  <p>For example, you give someone 1 dollar or 1 euro. You can forgive them, you tolerate it. Your hinge loss is 0 for lending someone 1d dollar. However, if you give them 10 dollars or 100, you will ask them to refund you ASAP because you can't tolerate that much loss!</p>
  <p><a href="https://i.sstatic.net/Nj05a.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Nj05a.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-24T03:03:25.053Z
Answers.Text: Hi. Thanks for trying to contribute, but you should at least try to address the actual question.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-24T13:30:49.107Z
Answers.Text: Your information regarding the hinge loss is interesting, but, as @nbro said, it doesn't answer my question.
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-24T00:41:07.560Z
Answers.LastActivityDate: 2020-11-24T00:41:07.560Z
Answers.UserDisplayName: ombk
Body: <p>I am currently studying the textbook <em>Neural Networks and Deep Learning</em> by Charu C. Aggarwal. Chapter <strong>1.2.1.2 Relationship with Support Vector Machines</strong> says the following:</p>
  <blockquote>
  <p>The perceptron criterion is a shifted version of the hinge-loss used in support vector machines (see Chapter 2). The hinge loss looks even more similar to the zero-one loss criterion of Equation 1.7, and is defined as follows:
  <span class="math-container">$$L_i^{svm} = \max\{ 1 - y_i(\overline{W} \cdot \overline{X}_i), 0 \} \tag{1.9}$$</span>
  Note that the perceptron does not keep the constant term of <span class="math-container">$1$</span> on the right-hand side of Equation 1.7, whereas the hinge loss keeps this constant within the maximization function. This change does not affect the algebraic expression for the gradient, but it does change which points are lossless and should not cause an update. The relationship between the
  perceptron criterion and the hinge loss is shown in Figure 1.6. This similarity becomes particularly evident when the perceptron updates of Equation 1.6 are rewritten as follows:
  <span class="math-container">$$\overline{W} \Leftarrow \overline{W} + \alpha \sum_{(\overline{X}, y) \in S^+} y \overline{X} \tag{1.10}$$</span>
  Here, <span class="math-container">$S^+$</span> is defined as the set of all misclassified training points <span class="math-container">$\overline{X} \in S$</span> that satisfy the condition <span class="math-container">$y(\overline{W} \cdot \overline{X}) &lt; 0$</span>. This update seems to look somewhat different from the perceptron, because the perceptron uses the error <span class="math-container">$E(\overline{X})$</span> for the update, which is replaced with <span class="math-container">$y$</span> in the update above. A key point is that the (integer) error value <span class="math-container">$E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$</span> can never be <span class="math-container">$0$</span> for misclassified points in <span class="math-container">$S^+$</span>. Therefore, we have <span class="math-container">$E(\overline{X}) = 2y$</span> <em>for misclassified points</em>, and <span class="math-container">$E(X)$</span> can be replaced with <span class="math-container">$y$</span> in the updates after absorbing the factor of <span class="math-container">$2$</span> within the learning rate.</p>
  </blockquote>
  <p>Equation 1.6 is as follows:</p>
  <blockquote>
  <p><span class="math-container">$$\overline{W} \Leftarrow \overline{W} + \alpha \sum_{\overline{X} \in S} E(\overline{X})\overline{X}, \tag{1.6}$$</span>
  where <span class="math-container">$S$</span> is a randomly chosen subset of training points, <span class="math-container">$\overline{X} = [x_1, \dots, x_d]$</span> is a data instance (vector of <span class="math-container">$d$</span> feature variables), <span class="math-container">$\overline{W} = [w_1, \dots, w_d]$</span> are the weights, <span class="math-container">$\alpha$</span> is the learning rate, and <span class="math-container">$E(\overline{X}) = (y - \hat{y})$</span> is an error value, where <span class="math-container">$\hat{y} = \text{sign}\{ \overline{W} \cdot \overline{X} \}$</span> is the prediction and <span class="math-container">$y$</span> is the observed value of the binary class variable.</p>
  </blockquote>
  <p>Equation 1.7 is as follows:</p>
  <blockquote>
  <p><span class="math-container">$$L_i^{(0/1)} = \dfrac{1}{2} (y_i - \text{sign}\{ \overline{W} \cdot \overline{X_i} \})^2 = 1 - y_i \cdot \text{sign} \{ \overline{W} \cdot \overline{X_i} \} \tag{1.7}$$</span></p>
  </blockquote>
  <p>And figure 1.6 is as follows:</p>
  <blockquote>
  <p><a href="https://i.sstatic.net/nmMyU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/nmMyU.png" alt="enter image description here" /></a></p>
  </blockquote>
  <p>Figure 1.6 looks unclear to me. What is figure 1.6 showing, and how is it relevant to the point that the author is trying to make?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-24T01:22:13.697Z
Comments.Text: I was also confused about this specific part of the book and came accross the following video explaining hinge loss. Hope it might help: https://www.youtube.com/watch?v=PM2MSAYmzXM
Comments.UserDisplayName: user45643
Tags: <objective-functions><support-vector-machine><perceptron><binary-classification><hinge-loss>
Title: How should we interpret this figure that relates the perceptron criterion and the hinge loss?
UserDisplayName: The Pointer
ThreadText: # How should we interpret this figure that relates the perceptron criterion and the hinge loss?
  
  (Asked by: username_0 on 2020-11-21)
  
  I am currently studying the textbook *Neural Networks and Deep Learning* by Charu C. Aggarwal. Chapter **1.2.1.2 Relationship with Support Vector Machines** says the following:
  
  > The perceptron criterion is a shifted version of the hinge-loss used in support vector machines (see Chapter 2). The hinge loss looks even more similar to the zero-one loss criterion of Equation 1.7, and is defined as follows:
  > $$L\_i^{svm} = \max\{ 1 - y\_i(\overline{W} \cdot \overline{X}\_i), 0 \} \tag{1.9}$$
  > Note that the perceptron does not keep the constant term of $1$ on the right-hand side of Equation 1.7, whereas the hinge loss keeps this constant within the maximization function. This change does not affect the algebraic expression for the gradient, but it does change which points are lossless and should not cause an update. The relationship between the
  > perceptron criterion and the hinge loss is shown in Figure 1.6. This similarity becomes particularly evident when the perceptron updates of Equation 1.6 are rewritten as follows:
  > $$\overline{W} \Leftarrow \overline{W} + \alpha \sum\_{(\overline{X}, y) \in S^+} y \overline{X} \tag{1.10}$$
  > Here, $S^+$ is defined as the set of all misclassified training points $\overline{X} \in S$ that satisfy the condition $y(\overline{W} \cdot \overline{X}) < 0$. This update seems to look somewhat different from the perceptron, because the perceptron uses the error $E(\overline{X})$ for the update, which is replaced with $y$ in the update above. A key point is that the (integer) error value $E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$ can never be $0$ for misclassified points in $S^+$. Therefore, we have $E(\overline{X}) = 2y$ *for misclassified points*, and $E(X)$ can be replaced with $y$ in the updates after absorbing the factor of $2$ within the learning rate.
  
  Equation 1.6 is as follows:
  
  > $$\overline{W} \Leftarrow \overline{W} + \alpha \sum\_{\overline{X} \in S} E(\overline{X})\overline{X}, \tag{1.6}$$
  > where $S$ is a randomly chosen subset of training points, $\overline{X} = [x\_1, \dots, x\_d]$ is a data instance (vector of $d$ feature variables), $\overline{W} = [w\_1, \dots, w\_d]$ are the weights, $\alpha$ is the learning rate, and $E(\overline{X}) = (y - \hat{y})$ is an error value, where $\hat{y} = \text{sign}\{ \overline{W} \cdot \overline{X} \}$ is the prediction and $y$ is the observed value of the binary class variable.
  
  Equation 1.7 is as follows:
  
  > $$L\_i^{(0/1)} = \dfrac{1}{2} (y\_i - \text{sign}\{ \overline{W} \cdot \overline{X\_i} \})^2 = 1 - y\_i \cdot \text{sign} \{ \overline{W} \cdot \overline{X\_i} \} \tag{1.7}$$
  
  And figure 1.6 is as follows:
  
  > [![enter image description here](https://i.sstatic.net/nmMyU.png)](https://i.sstatic.net/nmMyU.png)
  
  Figure 1.6 looks unclear to me. What is figure 1.6 showing, and how is it relevant to the point that the author is trying to make?
  
  **Tags:** objective-functions, support-vector-machine, perceptron, binary-classification, hinge-loss
  > **Comments:**
  > **username_1**: I was also confused about this specific part of the book and came accross the following video explaining hinge loss. Hope it might help: https://www.youtube.com/watch?v=PM2MSAYmzXM  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-11-05. Score: 0
  
  Figure 1.6 is depicting that the hinge-loss used in SVMs (Eqn 1.9):
  
  $$L\_i^{svm} = \text{max}\{1 - y\_i (\overline{W} \cdot \overline{X}\_i, 0\} \tag{1.9}$$
  
  is a shifted version (+1 to the right) of the loss function (Eqn 1.8) used to optimize the perceptron criterion (see footnote (1)):
  
  $$L\_i = \text{max}\{-y\_i (\overline{W} \cdot \overline{X}\_i), 0\} \tag{1.8}$$
  
  Using Figure 1.6 and the accompanying discussion in Section 1.2.1.2, the author (Charu C. Aggarwal) is trying to show that if you make the implicit training process of the perceptron explicit (with an objective function, in his case, a loss function defined by Eqn 1.8), then one realises (in his words):
  
  > that the perceptron is fundamentally not very different from well-known machine learning algorithms like the support vector machine in spite of its different origins.
  
  ### Footnotes
  
  (1) The *perceptron criterion* was introduced in Section 3.5.1 of [Neural Networks for Pattern Recognition](https://www.academia.edu/download/55593223/_Christopher_M._Bishop__Neural_Networks_for_Patterb-ok.org.pdf) by Christopher Bishop as a *continuous*, piecewise linear error function given by:
  
  $$E^{\text{perc}}(\bf{w}) = - \sum\_{\phi^n \in \mathcal{m}} w^T (\phi^n t^n)$$
  
  Bishop suggested that this objective function is minimised when training a perceptron. Aggarwal references this work as an example of how the perceptron training process implicitly minimises this error function, using it as a segue into his own derivation of Eqn 1.8 before encouraging the reader to check that the gradient of Eqn 1.8 leads to the perceptron update (i.e. $\overline{W} \Leftarrow \overline{W} - \alpha \nabla\_{W} L\_i$).
  
  
  ---
  
  ### Answer by username_3 on 2020-11-24. Score: -2
  
  I can't fully explain the part because I forgot what it talks about.
  
  However, regarding the hinge loss, it is basically allowing your SVM to tolerate misclassifications without increasing the cost function.
  
  For example, you give someone 1 dollar or 1 euro. You can forgive them, you tolerate it. Your hinge loss is 0 for lending someone 1d dollar. However, if you give them 10 dollars or 100, you will ask them to refund you ASAP because you can't tolerate that much loss!
  
  [![enter image description here](https://i.sstatic.net/Nj05a.png)](https://i.sstatic.net/Nj05a.png)
  
  > **Comments:**
  > **username_4**: Hi. Thanks for trying to contribute, but you should at least try to address the actual question.  
  > **username_0**: Your information regarding the hinge loss is interesting, but, as @nbro said, it doesn't answer my question.  
  
  ---
  
  