Answers.Body: <p>It depends on your hypothesis <span class="math-container">$h$</span>. The author of the original article compares the dot product with a threshold:</p>
  <p><a href="https://i.sstatic.net/0RdSq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0RdSq.png" alt="enter image description here" /></a></p>
  <p>So for a binary classification problem <span class="math-container">$h$</span> can be defined as follows:</p>
  <p><span class="math-container">$$
  h  = \begin{cases}
        1 &amp; \text{if $f&gt;z$}\\
        0 &amp; \text{otherwise}
      \end{cases}
  $$</span></p>
  <p>That is, <span class="math-container">$\hat y_i$</span> is your prediction and <span class="math-container">$\hat y_i = h(x_i)$</span>, <span class="math-container">$y_i$</span> is a real label and <span class="math-container">$x_i$</span> is a sample.</p>
  <p>Finally, you can update your weghts <span class="math-container">$w_n = w_n + \eta(y_i - \hat y_i)x_i$</span>, where <span class="math-container">$n$</span> is a number of the weight and <span class="math-container">$i$</span> denotes a number of the label/sample pair.</p>
  <p>Depending on how you define your hypothesis you will have a different optimization algorithm. Take a look at <a href="https://ai.stackexchange.com/questions/28235/where-does-the-so-called-loss-loss-function-fit-into-the-idea-of-a-percept/28283#28283">this answer</a> for more details.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-23T22:38:06.940Z
Answers.Text: Thank you. How do I get $(y_i - \hat y_i)x_i$ from this hypothesis \begin{align}
  \hat{y} &= sign(\mathbf{w} \cdot \mathbf{x} + b) \\
  &= sign({w}_{1}{x}_{1}+{w}_{2}{x}_{2} + ... + w_nx_n + b) \\
  \end{align}
Answers.UserDisplayName: JJJohn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-24T01:40:58.733Z
Answers.Text: [sign](https://en.wikipedia.org/wiki/Sign_(mathematics)) extracts a sign of a number. According to the article, it is defined exactly as I described: $sign(\mathbf{w} \cdot \mathbf{x} + b) = \begin{cases}
        1 & \text{if $\mathbf{w} \cdot \mathbf{x} + b>0$}\\
        0 & \text{otherwise}
      \end{cases}$
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-22T15:36:58.823Z
Answers.LastActivityDate: 2021-06-22T15:50:42.320Z
Answers.LastEditDate: 2021-06-22T15:50:42.320Z
Answers.UserDisplayName: Aray Karjauv
Answers.Body: <p>Your reference's training algo is called error correction rule or simply perceptron 0-1 rule which doesn't have the usual mean squared error (MSE) or cross entropy (CE) loss function associated with LMS/delta training rules leveraging various forms of gradient descent (GD) methods because its activation function is the non-differentiable Heaviside sign function breaking any GD applicability via chain rule to update each weight. Yet instead it could be <em>approximately</em> viewed as having MSE loss function optimized with stochastic gradient descent (SGD) method iterating over every training sample to update each weight incrementally following your referenced perceptron rule.</p>
  <p>This can be intuited from simply replacing your sign activation function with a very similarly shaped differentiable function such as sigmoid, then your concerned 0-1 training rule can be derived just as the usual delta rule reflected in the Widrow-Hoff SGD version algo or the popular MLP backpropagation's SGD version algo to update any weight parameter, and better yet in your case, the derivative of activation function is approximately 1 since the activation function is assumed to be very like the sign function as intuited above.</p>
  <p>Of course this is just some intuition, even for linearly separable dataset we cannot assure your concerned rule would converge at all. Fortunately there was <a href="https://en.wikipedia.org/wiki/Perceptron#Convergence_of_one_perceptron_on_a_linearly_separable_dataset" rel="nofollow noreferrer">Perceptron convergence theorem</a> proved by Rosenblatt et al to affirm such convergence in the cases data is linearly separable. Even in non-separable cases there is no way for a single perceptron to converge, however, we still have <a href="https://en.wikipedia.org/wiki/Perceptron#Perceptron_cycling_theorem" rel="nofollow noreferrer">Perceptron cycling theorem</a> to give some upper bound estimate for the weight vector norm.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-11T07:57:52.710Z
Answers.LastActivityDate: 2023-11-11T08:22:33.457Z
Answers.LastEditDate: 2023-11-11T08:22:33.457Z
Answers.UserDisplayName: cinch
Body: <p>A <a href="https://towardsdatascience.com/6-steps-to-write-any-machine-learning-algorithm-from-scratch-perceptron-case-study-335f638a70f3" rel="nofollow noreferrer">post</a> gives a formula for perceptron to update weights</p>
  <p><a href="https://i.sstatic.net/qgovN.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qgovN.jpg" alt="enter image description here" /></a></p>
  <p>I understand almost all the parts of it, except for the part <span class="math-container">$(y_i - \hat y_i)x_i$</span> where does it come from? Is it the gradient of some kind of loss function? If yes, what is the definition of the loss function?</p>
  <p>The OP seems doesn't give the hypothesis, so that <span class="math-container">$\hat y_i = h(x_i)$</span></p>
  <p>However, this hypothesis seems prevalent</p>
  <p><span class="math-container">\begin{align}
  \hat{y} &amp;= sign(\mathbf{w} \cdot \mathbf{x} + b) \tag{1}\\
  &amp;= sign({w}_{1}{x}_{1}+{w}_{2}{x}_{2} + ... + w_nx_n + b) \\
  \end{align}</span></p>
  <p>where</p>
  <p><span class="math-container">$$
  sign(z) = 
  \begin{cases}
  1, &amp; z \ge 0 \\
  -1, &amp; z &lt; 0
  \end{cases}
  $$</span></p>
  <p>How do I get <span class="math-container">$(y_i - \hat y_i)x_i$</span> from function (1)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-24T23:28:07.080Z
Comments.Text: I'm only adding this comment because you asked, "Is it the gradient of some kind of loss function?"  The direct answer is "No... It is, in itself, a simple loss function."  I just want to make sure that you recognize that the author seems to be presenting a simplified approach and not what we tend to use in a deep learning neural network.  More typically we will be performing backpropagation with partial differentials to determine how the weights and biases are adjusted.  Andrew Ng has a very, very good video explaining this that simplifies the math significantly if you search around.
Comments.UserDisplayName: David Hoelzer
Tags: <machine-learning><perceptron>
Title: Is $(y_i - \hat y_i)x_i$, part of the formula for updating weights for perceptron, the gradient of some kind of loss function?
UserDisplayName: JJJohn
ThreadText: # Is $(y_i - \hat y_i)x_i$, part of the formula for updating weights for perceptron, the gradient of some kind of loss function?
  
  (Asked by: username_0 on 2021-06-22)
  
  A [post](https://towardsdatascience.com/6-steps-to-write-any-machine-learning-algorithm-from-scratch-perceptron-case-study-335f638a70f3) gives a formula for perceptron to update weights
  
  [![enter image description here](https://i.sstatic.net/qgovN.jpg)](https://i.sstatic.net/qgovN.jpg)
  
  I understand almost all the parts of it, except for the part $(y\_i - \hat y\_i)x\_i$ where does it come from? Is it the gradient of some kind of loss function? If yes, what is the definition of the loss function?
  
  The OP seems doesn't give the hypothesis, so that $\hat y\_i = h(x\_i)$
  
  However, this hypothesis seems prevalent
  
  \begin{align}
  \hat{y} &= sign(\mathbf{w} \cdot \mathbf{x} + b) \tag{1}\\
  &= sign({w}\_{1}{x}\_{1}+{w}\_{2}{x}\_{2} + ... + w\_nx\_n + b) \\
  \end{align}
  
  where
  
  $$
  sign(z) =
  \begin{cases}
  1, & z \ge 0 \\
  -1, & z < 0
  \end{cases}
  $$
  
  How do I get $(y\_i - \hat y\_i)x\_i$ from function (1)
  
  **Tags:** machine-learning, perceptron
  > **Comments:**
  > **username_1**: I'm only adding this comment because you asked, "Is it the gradient of some kind of loss function?"  The direct answer is "No... It is, in itself, a simple loss function."  I just want to make sure that you recognize that the author seems to be presenting a simplified approach and not what we tend to use in a deep learning neural network.  More typically we will be performing backpropagation with partial differentials to determine how the weights and biases are adjusted.  Andrew Ng has a very, very good video explaining this that simplifies the math significantly if you search around.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-06-22. Score: 0
  
  It depends on your hypothesis $h$. The author of the original article compares the dot product with a threshold:
  
  [![enter image description here](https://i.sstatic.net/0RdSq.png)](https://i.sstatic.net/0RdSq.png)
  
  So for a binary classification problem $h$ can be defined as follows:
  
  $$
  h = \begin{cases}
  1 & \text{if $f>z$}\\
  0 & \text{otherwise}
  \end{cases}
  $$
  
  That is, $\hat y\_i$ is your prediction and $\hat y\_i = h(x\_i)$, $y\_i$ is a real label and $x\_i$ is a sample.
  
  Finally, you can update your weghts $w\_n = w\_n + \eta(y\_i - \hat y\_i)x\_i$, where $n$ is a number of the weight and $i$ denotes a number of the label/sample pair.
  
  Depending on how you define your hypothesis you will have a different optimization algorithm. Take a look at [this answer](https://ai.stackexchange.com/questions/28235/where-does-the-so-called-loss-loss-function-fit-into-the-idea-of-a-percept/28283#28283) for more details.
  
  > **Comments:**
  > **username_0**: Thank you. How do I get $(y_i - \hat y_i)x_i$ from this hypothesis \begin{align} \hat{y} &= sign(\mathbf{w} \cdot \mathbf{x} + b) \\ &= sign({w}_{1}{x}_{1}+{w}_{2}{x}_{2} + ... + w_nx_n + b) \\ \end{align}  
  > **username_2**: [sign](https://en.wikipedia.org/wiki/Sign_(mathematics)) extracts a sign of a number. According to the article, it is defined exactly as I described: $sign(\mathbf{w} \cdot \mathbf{x} + b) = \begin{cases}       1 & \text{if $\mathbf{w} \cdot \mathbf{x} + b>0$}\\       0 & \text{otherwise}     \end{cases}$  
  
  ---
  
  ### Answer by username_3 on 2023-11-11. Score: 0
  
  Your reference's training algo is called error correction rule or simply perceptron 0-1 rule which doesn't have the usual mean squared error (MSE) or cross entropy (CE) loss function associated with LMS/delta training rules leveraging various forms of gradient descent (GD) methods because its activation function is the non-differentiable Heaviside sign function breaking any GD applicability via chain rule to update each weight. Yet instead it could be *approximately* viewed as having MSE loss function optimized with stochastic gradient descent (SGD) method iterating over every training sample to update each weight incrementally following your referenced perceptron rule.
  
  This can be intuited from simply replacing your sign activation function with a very similarly shaped differentiable function such as sigmoid, then your concerned 0-1 training rule can be derived just as the usual delta rule reflected in the Widrow-Hoff SGD version algo or the popular MLP backpropagation's SGD version algo to update any weight parameter, and better yet in your case, the derivative of activation function is approximately 1 since the activation function is assumed to be very like the sign function as intuited above.
  
  Of course this is just some intuition, even for linearly separable dataset we cannot assure your concerned rule would converge at all. Fortunately there was [Perceptron convergence theorem](https://en.wikipedia.org/wiki/Perceptron#Convergence_of_one_perceptron_on_a_linearly_separable_dataset) proved by Rosenblatt et al to affirm such convergence in the cases data is linearly separable. Even in non-separable cases there is no way for a single perceptron to converge, however, we still have [Perceptron cycling theorem](https://en.wikipedia.org/wiki/Perceptron#Perceptron_cycling_theorem) to give some upper bound estimate for the weight vector norm.
  
  
  ---
  
  