Answers.Body: <ol>
  <li>It is important to note that the exact statement is the eqation given below can never be 0 for misclassified points in <span class="math-container">$ S^+$</span>
  <span class="math-container">$$ E(X) = (y - \text{sign}\{\overline{W} \cdot \overline{X}\}) $$</span></li>
  <li>And <span class="math-container">$S+$</span> is defined as the set of all misclassified training points <span class="math-container">$X \in S$</span> that satisfy the condition <span class="math-container">$y(\overline{W} \cdot \overline{X})&lt;0 $</span> which means that <span class="math-container">$y$</span> and <span class="math-container">$ (\overline{W} \cdot \overline{X}) $</span> will have different signs i.e if <span class="math-container">$y &lt; 0$</span> then <span class="math-container">$(\overline{W} \cdot \overline{X}) &gt; 0$</span> and vice versa.</li>
  <li>Finally the condition on <span class="math-container">$S^+$</span> constraints the the <span class="math-container">$ E(X)$</span> to these values when <span class="math-container">$ y &lt; 0$</span> and <span class="math-container">$(\overline{W} \cdot \overline{X}) &gt; 0$</span>
  <span class="math-container">$$ E(X) = (-1 - \text{sign}(+ve)) = -2 $$</span>
  or when <span class="math-container">$ y &gt; 0$</span> and <span class="math-container">$(\overline{W} \cdot \overline{X}) &lt; 0$</span>
  <span class="math-container">$$ E(X) = (1 - \text{sign}(-ve)) = 2 $$</span></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-21T19:39:37.663Z
Answers.LastActivityDate: 2021-02-07T16:04:54.170Z
Answers.LastEditDate: 2021-02-07T16:04:54.170Z
Answers.UserDisplayName: ssh
Body: <p>I am currently studying the textbook <em>Neural Networks and Deep Learning</em> by Charu C. Aggarwal. Chapter <strong>1.2.1.2 Relationship with Support Vector Machines</strong> says the following:</p>
  <blockquote>
  <p>The perceptron criterion is a shifted version of the hinge-loss used in support vector machines (see Chapter 2). The hinge loss looks even more similar to the zero-one loss criterion of Equation 1.7, and is defined as follows:
  <span class="math-container">$$L_i^{svm} = \max\{ 1 - y_i(\overline{W} \cdot \overline{X}_i), 0 \} \tag{1.9}$$</span>
  Note that the perceptron does not keep the constant term of <span class="math-container">$1$</span> on the right-hand side of Equation 1.7, whereas the hinge loss keeps this constant within the maximization function. This change does not affect the algebraic expression for the gradient, but it does change which points are lossless and should not cause an update. The relationship between the
  perceptron criterion and the hinge loss is shown in Figure 1.6. This similarity becomes
  particularly evident when the perceptron updates of Equation 1.6 are rewritten as follows:
  <span class="math-container">$$\overline{W} \Leftarrow \overline{W} + \alpha \sum_{(\overline{X}, y) \in S^+} y \overline{X} \tag{1.10}$$</span>
  Here, <span class="math-container">$S^+$</span> is defined as the set of all misclassified training points <span class="math-container">$\overline{X} \in S$</span> that satisfy the condition <span class="math-container">$y(\overline{W} \cdot \overline{X}) &lt; 0$</span>. This update seems to look somewhat different from the perceptron, because the perceptron uses the error <span class="math-container">$E(\overline{X})$</span> for the update, which is replaced with <span class="math-container">$y$</span> in the update above. A key point is that the (integer) error value <span class="math-container">$E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$</span> can never be <span class="math-container">$0$</span> for misclassified points in <span class="math-container">$S^+$</span>. Therefore, we have <span class="math-container">$E(\overline{X}) = 2y$</span> <em>for misclassified points</em>, and <span class="math-container">$E(X)$</span> can be replaced with <span class="math-container">$y$</span> in the updates after absorbing the factor of <span class="math-container">$2$</span> within the learning rate.</p>
  </blockquote>
  <p>Equation 1.7 is as follows:</p>
  <blockquote>
  <p><span class="math-container">$$L_i^{(0/1)} = \dfrac{1}{2} (y_i - \text{sign}\{ \overline{W} \cdot \overline{X_i} \})^2 = 1 - y_i \cdot \text{sign} \{ \overline{W} \cdot \overline{X_i} \} \tag{1.7}$$</span></p>
  </blockquote>
  <p>And figure 1.6 is as follows:</p>
  <blockquote>
  <p><a href="https://i.sstatic.net/nmMyU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/nmMyU.png" alt="enter image description here" /></a></p>
  </blockquote>
  <p>It is said that we are dealing with the case of binary classification, where <span class="math-container">$y \in \{ -1, +1 \}$</span>. But the author claims that <span class="math-container">$E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$</span>, which doesn't include the case of <span class="math-container">$0$</span>. So shouldn't the <span class="math-container">$\{ -2, +2 \}$</span> in <span class="math-container">$E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$</span> be <span class="math-container">$\{ -2, 0, +2 \}$</span>?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-21T14:33:18.543Z
Comments.Text: The author is saying that $E(X)$ can never be zero for misclassified points, so I am not sure I understand your question/concerns. (Btw, I didn't really read that excerpt carefully).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-21T14:38:30.857Z
Comments.Text: @nbro But why can it never be zero for misclassified points? As I said, $y \in \{ -1, +1 \}$, so I don't see why $E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \})$ cannot be zero.
Comments.UserDisplayName: The Pointer
Tags: <objective-functions><support-vector-machine><perceptron><binary-classification>
Title: Why doesn't the set $\{ -2, +2 \}$ in $E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$ include $0$?
UserDisplayName: The Pointer
ThreadText: # Why doesn't the set $\{ -2, +2 \}$ in $E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$ include $0$?
  
  (Asked by: username_0 on 2020-11-21)
  
  I am currently studying the textbook *Neural Networks and Deep Learning* by Charu C. Aggarwal. Chapter **1.2.1.2 Relationship with Support Vector Machines** says the following:
  
  > The perceptron criterion is a shifted version of the hinge-loss used in support vector machines (see Chapter 2). The hinge loss looks even more similar to the zero-one loss criterion of Equation 1.7, and is defined as follows:
  > $$L\_i^{svm} = \max\{ 1 - y\_i(\overline{W} \cdot \overline{X}\_i), 0 \} \tag{1.9}$$
  > Note that the perceptron does not keep the constant term of $1$ on the right-hand side of Equation 1.7, whereas the hinge loss keeps this constant within the maximization function. This change does not affect the algebraic expression for the gradient, but it does change which points are lossless and should not cause an update. The relationship between the
  > perceptron criterion and the hinge loss is shown in Figure 1.6. This similarity becomes
  > particularly evident when the perceptron updates of Equation 1.6 are rewritten as follows:
  > $$\overline{W} \Leftarrow \overline{W} + \alpha \sum\_{(\overline{X}, y) \in S^+} y \overline{X} \tag{1.10}$$
  > Here, $S^+$ is defined as the set of all misclassified training points $\overline{X} \in S$ that satisfy the condition $y(\overline{W} \cdot \overline{X}) < 0$. This update seems to look somewhat different from the perceptron, because the perceptron uses the error $E(\overline{X})$ for the update, which is replaced with $y$ in the update above. A key point is that the (integer) error value $E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$ can never be $0$ for misclassified points in $S^+$. Therefore, we have $E(\overline{X}) = 2y$ *for misclassified points*, and $E(X)$ can be replaced with $y$ in the updates after absorbing the factor of $2$ within the learning rate.
  
  Equation 1.7 is as follows:
  
  > $$L\_i^{(0/1)} = \dfrac{1}{2} (y\_i - \text{sign}\{ \overline{W} \cdot \overline{X\_i} \})^2 = 1 - y\_i \cdot \text{sign} \{ \overline{W} \cdot \overline{X\_i} \} \tag{1.7}$$
  
  And figure 1.6 is as follows:
  
  > [![enter image description here](https://i.sstatic.net/nmMyU.png)](https://i.sstatic.net/nmMyU.png)
  
  It is said that we are dealing with the case of binary classification, where $y \in \{ -1, +1 \}$. But the author claims that $E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$, which doesn't include the case of $0$. So shouldn't the $\{ -2, +2 \}$ in $E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$ be $\{ -2, 0, +2 \}$?
  
  **Tags:** objective-functions, support-vector-machine, perceptron, binary-classification
  > **Comments:**
  > **username_1**: The author is saying that $E(X)$ can never be zero for misclassified points, so I am not sure I understand your question/concerns. (Btw, I didn't really read that excerpt carefully).  
  > **username_0**: @nbro But why can it never be zero for misclassified points? As I said, $y \in \{ -1, +1 \}$, so I don't see why $E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \})$ cannot be zero.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-11-21. Score: 2 | Accepted answer
  
  1. It is important to note that the exact statement is the eqation given below can never be 0 for misclassified points in $ S^+$
     $$ E(X) = (y - \text{sign}\{\overline{W} \cdot \overline{X}\}) $$
  2. And $S+$ is defined as the set of all misclassified training points $X \in S$ that satisfy the condition $y(\overline{W} \cdot \overline{X})<0 $ which means that $y$ and $ (\overline{W} \cdot \overline{X}) $ will have different signs i.e if $y < 0$ then $(\overline{W} \cdot \overline{X}) > 0$ and vice versa.
  3. Finally the condition on $S^+$ constraints the the $ E(X)$ to these values when $ y < 0$ and $(\overline{W} \cdot \overline{X}) > 0$
     $$ E(X) = (-1 - \text{sign}(+ve)) = -2 $$
     or when $ y > 0$ and $(\overline{W} \cdot \overline{X}) < 0$
     $$ E(X) = (1 - \text{sign}(-ve)) = 2 $$
  
  
  ---
  
  