Answers.Body: <p>I will tell you my knowledge, correct me if I am wrong.</p>
  <p>Perceptron Learning Algorithm (PLA) is a simple method to solve the <strong>binary classification</strong> problem.</p>
  <p>Define a function:</p>
  <p><span class="math-container">$$
  f_w(x) = w^Tx + b
  $$</span></p>
  <p>where <span class="math-container">$x \in \mathbb{R}^n$</span> is an input vector that contains data points and <span class="math-container">$w$</span> is a vector with the same dimension as <span class="math-container">$x$</span> which present for the parameters of our model.</p>
  <p>Call <span class="math-container">$y=label(x)=\{1,-1\}$</span> where <span class="math-container">$1$</span> and <span class="math-container">$-1$</span> are the label of each <span class="math-container">$x$</span> vector.</p>
  <p>The PLA will predict a class like this:</p>
  <p><span class="math-container">$$
  y=label(x)=sgn(f_w(x))=sgn(w^Tx+b)
  $$</span></p>
  <p>(The definition of sgn function can be found in <a href="https://en.wikipedia.org/wiki/Sign_function" rel="nofollow noreferrer">this wiki</a>)</p>
  <p>We can understand that PLA tries to define a line (in 2D, or a plane in 3D, and hyperplane in more than 3 dimensions coordinate, I will assume it in 2D from now on) which separate our data into two areas. So how can we find that line? Just like every other machine learning problems, define a cost function, then optimize the parameters to have the smallest cost value.</p>
  <p>Now, let define the cost function first, you can see that if a data point lies in the correct area, <span class="math-container">$y$</span> and <span class="math-container">$f(x)$</span> have the same sign, which means <span class="math-container">$y(w^Tx+b) &gt; 0$</span> and otherwise. Similar to your example, I will define:
  <span class="math-container">$$
  g(x)=y(w^Tx+b)
  $$</span></p>
  <p>We ignore all the points in the safe zone (<span class="math-container">$g(x)&gt;0$</span>), only update to rotate or move the line to adapt with the misclassified points (<span class="math-container">$g(x)\le 0$</span>), here, you can understand why we only update if <span class="math-container">$g(x)\le0$</span>.</p>
  <p>We need to define a cost function to minimize it, so our cost function will become:
  <span class="math-container">$$
  L(w)=\displaystyle\sum_{x_i\in U}(-y_i(w^Tx_i+b))
  $$</span>
  where</p>
  <ul>
  <li><span class="math-container">$U$</span>is the set of the misclassified points</li>
  <li><span class="math-container">$y_i$</span> is the label of data point <span class="math-container">$i$</span>-th</li>
  <li><span class="math-container">$x_i$</span> is the <span class="math-container">$i$</span>-th data vector</li>
  <li><span class="math-container">$w$</span> and <span class="math-container">$b$</span> is parameters of our model</li>
  </ul>
  <p>For each data point, we have the derivative is
  <span class="math-container">$$
  \frac{\partial L}{\partial w} = -y_ix_i \\
  \frac{\partial L}{\partial b} = -y_i
  $$</span></p>
  <p>Finally, update them by Stochastic gradient descent (SGD), we get:
  <span class="math-container">$$
  w = w - \frac{\partial L}{\partial w} = w + y_ix_i \\
  b = b - \frac{\partial L}{\partial b} = b + y_i
  $$</span></p>
  <p>For your last question, notice that the weight and bias changed from <span class="math-container">$4$</span>-th updated, so we have:
  <span class="math-container">$$
  y_5 = 1, x_5 = (4,8), w = (-2,-2), b = -2 \\
  \Rightarrow g_5 = +1 \times (4\times(-2) + 8\times(-2)+ (-2)) = -8 -16 -2
  $$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-24T06:09:56.067Z
Answers.LastActivityDate: 2020-12-24T06:09:56.067Z
Answers.UserDisplayName: CuCaRot
Body: <p>I'm trying to understand the solution to question 4 of <a href="https://alex.smola.org/teaching/10-701-15/exam/midterm.pdf" rel="nofollow noreferrer">this</a> midterm paper.</p>
  <p>The question and solution is as follows:</p>
  <p><a href="https://i.sstatic.net/pdTDw.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pdTDw.png" alt="enter image description here" /></a></p>
  <p>I thought that the process for updating weights was:</p>
  <pre><code>error = target - guess
  new_weight = current_weight + (error)(input)
  </code></pre>
  <p>I do not understand for example, for number 2 below, how that sum is determined. For example, I want to understand whether to update the weight or not. The calculation is:</p>
  <pre><code>x1(w1) + x2(w2)
  (10)(1) + (10)(1) = 20
  20 &gt; 0, therefore update.
  </code></pre>
  <p>But the equation to obtain the same answer in the solution is:</p>
  <pre><code>1(10 + 10) 20
  20 &gt; 0, therefore update.
  </code></pre>
  <p>I understand that these two equations are essentially the same, but written differently. But for example, in step 5, what do the elements in g5 mean. What do the -8, -16 and -2 represent?</p>
  <p>p.s. I know in a previous (now deleted) post of mine, I asked a question related to the use of LaTeX instead for maths equations. If someone can show me a simple way to convert these equations online, I'm more than happy to use it. However, I'm unfamiliar with this software, so I need some sort of converter.</p>
  
Tags: <machine-learning><perceptron>
Title: What is the equation to update the weights in the perceptron algorithm?
UserDisplayName: Slowat_Kela
ThreadText: # What is the equation to update the weights in the perceptron algorithm?
  
  (Asked by: username_0 on 2020-12-23)
  
  I'm trying to understand the solution to question 4 of [this](https://alex.smola.org/teaching/10-701-15/exam/midterm.pdf) midterm paper.
  
  The question and solution is as follows:
  
  [![enter image description here](https://i.sstatic.net/pdTDw.png)](https://i.sstatic.net/pdTDw.png)
  
  I thought that the process for updating weights was:
  
  ```
  error = target - guess
  new_weight = current_weight + (error)(input)
  
  ```
  
  I do not understand for example, for number 2 below, how that sum is determined. For example, I want to understand whether to update the weight or not. The calculation is:
  
  ```
  x1(w1) + x2(w2)
  (10)(1) + (10)(1) = 20
  20 > 0, therefore update.
  
  ```
  
  But the equation to obtain the same answer in the solution is:
  
  ```
  1(10 + 10) 20
  20 > 0, therefore update.
  
  ```
  
  I understand that these two equations are essentially the same, but written differently. But for example, in step 5, what do the elements in g5 mean. What do the -8, -16 and -2 represent?
  
  p.s. I know in a previous (now deleted) post of mine, I asked a question related to the use of LaTeX instead for maths equations. If someone can show me a simple way to convert these equations online, I'm more than happy to use it. However, I'm unfamiliar with this software, so I need some sort of converter.
  
  **Tags:** machine-learning, perceptron
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-24. Score: 1 | Accepted answer
  
  I will tell you my knowledge, correct me if I am wrong.
  
  Perceptron Learning Algorithm (PLA) is a simple method to solve the **binary classification** problem.
  
  Define a function:
  
  $$
  f\_w(x) = w^Tx + b
  $$
  
  where $x \in \mathbb{R}^n$ is an input vector that contains data points and $w$ is a vector with the same dimension as $x$ which present for the parameters of our model.
  
  Call $y=label(x)=\{1,-1\}$ where $1$ and $-1$ are the label of each $x$ vector.
  
  The PLA will predict a class like this:
  
  $$
  y=label(x)=sgn(f\_w(x))=sgn(w^Tx+b)
  $$
  
  (The definition of sgn function can be found in [this wiki](https://en.wikipedia.org/wiki/Sign_function))
  
  We can understand that PLA tries to define a line (in 2D, or a plane in 3D, and hyperplane in more than 3 dimensions coordinate, I will assume it in 2D from now on) which separate our data into two areas. So how can we find that line? Just like every other machine learning problems, define a cost function, then optimize the parameters to have the smallest cost value.
  
  Now, let define the cost function first, you can see that if a data point lies in the correct area, $y$ and $f(x)$ have the same sign, which means $y(w^Tx+b) > 0$ and otherwise. Similar to your example, I will define:
  $$
  g(x)=y(w^Tx+b)
  $$
  
  We ignore all the points in the safe zone ($g(x)>0$), only update to rotate or move the line to adapt with the misclassified points ($g(x)\le 0$), here, you can understand why we only update if $g(x)\le0$.
  
  We need to define a cost function to minimize it, so our cost function will become:
  $$
  L(w)=\displaystyle\sum\_{x\_i\in U}(-y\_i(w^Tx\_i+b))
  $$
  where
  
  * $U$is the set of the misclassified points
  * $y\_i$ is the label of data point $i$-th
  * $x\_i$ is the $i$-th data vector
  * $w$ and $b$ is parameters of our model
  
  For each data point, we have the derivative is
  $$
  \frac{\partial L}{\partial w} = -y\_ix\_i \\
  \frac{\partial L}{\partial b} = -y\_i
  $$
  
  Finally, update them by Stochastic gradient descent (SGD), we get:
  $$
  w = w - \frac{\partial L}{\partial w} = w + y\_ix\_i \\
  b = b - \frac{\partial L}{\partial b} = b + y\_i
  $$
  
  For your last question, notice that the weight and bias changed from $4$-th updated, so we have:
  $$
  y\_5 = 1, x\_5 = (4,8), w = (-2,-2), b = -2 \\
  \Rightarrow g\_5 = +1 \times (4\times(-2) + 8\times(-2)+ (-2)) = -8 -16 -2
  $$
  
  
  ---
  
  