<p>I will tell you my knowledge, correct me if I am wrong.</p>
<p>Perceptron Learning Algorithm (PLA) is a simple method to solve the <strong>binary classification</strong> problem.</p>
<p>Define a function:</p>
<p><span class="math-container">$$
f_w(x) = w^Tx + b
$$</span></p>
<p>where <span class="math-container">$x \in \mathbb{R}^n$</span> is an input vector that contains data points and <span class="math-container">$w$</span> is a vector with the same dimension as <span class="math-container">$x$</span> which present for the parameters of our model.</p>
<p>Call <span class="math-container">$y=label(x)=\{1,-1\}$</span> where <span class="math-container">$1$</span> and <span class="math-container">$-1$</span> are the label of each <span class="math-container">$x$</span> vector.</p>
<p>The PLA will predict a class like this:</p>
<p><span class="math-container">$$
y=label(x)=sgn(f_w(x))=sgn(w^Tx+b)
$$</span></p>
<p>(The definition of sgn function can be found in <a href="https://en.wikipedia.org/wiki/Sign_function" rel="nofollow noreferrer">this wiki</a>)</p>
<p>We can understand that PLA tries to define a line (in 2D, or a plane in 3D, and hyperplane in more than 3 dimensions coordinate, I will assume it in 2D from now on) which separate our data into two areas. So how can we find that line? Just like every other machine learning problems, define a cost function, then optimize the parameters to have the smallest cost value.</p>
<p>Now, let define the cost function first, you can see that if a data point lies in the correct area, <span class="math-container">$y$</span> and <span class="math-container">$f(x)$</span> have the same sign, which means <span class="math-container">$y(w^Tx+b) &gt; 0$</span> and otherwise. Similar to your example, I will define:
<span class="math-container">$$
g(x)=y(w^Tx+b)
$$</span></p>
<p>We ignore all the points in the safe zone (<span class="math-container">$g(x)&gt;0$</span>), only update to rotate or move the line to adapt with the misclassified points (<span class="math-container">$g(x)\le 0$</span>), here, you can understand why we only update if <span class="math-container">$g(x)\le0$</span>.</p>
<p>We need to define a cost function to minimize it, so our cost function will become:
<span class="math-container">$$
L(w)=\displaystyle\sum_{x_i\in U}(-y_i(w^Tx_i+b))
$$</span>
where</p>
<ul>
<li><span class="math-container">$U$</span>is the set of the misclassified points</li>
<li><span class="math-container">$y_i$</span> is the label of data point <span class="math-container">$i$</span>-th</li>
<li><span class="math-container">$x_i$</span> is the <span class="math-container">$i$</span>-th data vector</li>
<li><span class="math-container">$w$</span> and <span class="math-container">$b$</span> is parameters of our model</li>
</ul>
<p>For each data point, we have the derivative is
<span class="math-container">$$
\frac{\partial L}{\partial w} = -y_ix_i \\
\frac{\partial L}{\partial b} = -y_i
$$</span></p>
<p>Finally, update them by Stochastic gradient descent (SGD), we get:
<span class="math-container">$$
w = w - \frac{\partial L}{\partial w} = w + y_ix_i \\
b = b - \frac{\partial L}{\partial b} = b + y_i
$$</span></p>
<p>For your last question, notice that the weight and bias changed from <span class="math-container">$4$</span>-th updated, so we have:
<span class="math-container">$$
y_5 = 1, x_5 = (4,8), w = (-2,-2), b = -2 \\
\Rightarrow g_5 = +1 \times (4\times(-2) + 8\times(-2)+ (-2)) = -8 -16 -2
$$</span></p>

CC BY-SA 4.0
2020-12-24T06:09:56.067Z
2020-12-24T06:09:56.067Z
CuCaRot
<p>I'm trying to understand the solution to question 4 of <a href="https://alex.smola.org/teaching/10-701-15/exam/midterm.pdf" rel="nofollow noreferrer">this</a> midterm paper.</p>
<p>The question and solution is as follows:</p>
<p><a href="https://i.sstatic.net/pdTDw.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pdTDw.png" alt="enter image description here" /></a></p>
<p>I thought that the process for updating weights was:</p>
<pre><code>error = target - guess
new_weight = current_weight + (error)(input)
</code></pre>
<p>I do not understand for example, for number 2 below, how that sum is determined. For example, I want to understand whether to update the weight or not. The calculation is:</p>
<pre><code>x1(w1) + x2(w2)
(10)(1) + (10)(1) = 20
20 &gt; 0, therefore update.
</code></pre>
<p>But the equation to obtain the same answer in the solution is:</p>
<pre><code>1(10 + 10) 20
20 &gt; 0, therefore update.
</code></pre>
<p>I understand that these two equations are essentially the same, but written differently. But for example, in step 5, what do the elements in g5 mean. What do the -8, -16 and -2 represent?</p>
<p>p.s. I know in a previous (now deleted) post of mine, I asked a question related to the use of LaTeX instead for maths equations. If someone can show me a simple way to convert these equations online, I'm more than happy to use it. However, I'm unfamiliar with this software, so I need some sort of converter.</p>

<machine-learning><perceptron>
What is the equation to update the weights in the perceptron algorithm?
Slowat_Kela
# What is the equation to update the weights in the perceptron algorithm?

(Asked by: username_0 on 2020-12-23)

I'm trying to understand the solution to question 4 of this midterm paper.
The question and solution is as follows:

I thought that the process for updating weights was:
error = target - guess
new_weight = current_weight + (error)(input)

I do not understand for example, for number 2 below, how that sum is determined. For example, I want to understand whether to update the weight or not. The calculation is:
x1(w1) + x2(w2)
(10)(1) + (10)(1) = 20
20 > 0, therefore update.

But the equation to obtain the same answer in the solution is:
1(10 + 10) 20
20 > 0, therefore update.

I understand that these two equations are essentially the same, but written differently. But for example, in step 5, what do the elements in g5 mean. What do the -8, -16 and -2 represent?
p.s. I know in a previous (now deleted) post of mine, I asked a question related to the use of LaTeX instead for maths equations. If someone can show me a simple way to convert these equations online, I'm more than happy to use it. However, I'm unfamiliar with this software, so I need some sort of converter.

**Tags:** machine-learning, perceptron


---

## 1 answers

### Answer by username_1 on 2020-12-24. Score: 1 | Accepted answer

I will tell you my knowledge, correct me if I am wrong.

Perceptron Learning Algorithm (PLA) is a simple method to solve the **binary classification** problem.

Define a function:

$$
f\_w(x) = w^Tx + b
$$

where $x \in \mathbb{R}^n$ is an input vector that contains data points and $w$ is a vector with the same dimension as $x$ which present for the parameters of our model.

Call $y=label(x)=\{1,-1\}$ where $1$ and $-1$ are the label of each $x$ vector.

The PLA will predict a class like this:

$$
y=label(x)=sgn(f\_w(x))=sgn(w^Tx+b)
$$

(The definition of sgn function can be found in [this wiki](https://en.wikipedia.org/wiki/Sign_function))

We can understand that PLA tries to define a line (in 2D, or a plane in 3D, and hyperplane in more than 3 dimensions coordinate, I will assume it in 2D from now on) which separate our data into two areas. So how can we find that line? Just like every other machine learning problems, define a cost function, then optimize the parameters to have the smallest cost value.

Now, let define the cost function first, you can see that if a data point lies in the correct area, $y$ and $f(x)$ have the same sign, which means $y(w^Tx+b) > 0$ and otherwise. Similar to your example, I will define:
$$
g(x)=y(w^Tx+b)
$$

We ignore all the points in the safe zone ($g(x)>0$), only update to rotate or move the line to adapt with the misclassified points ($g(x)\le 0$), here, you can understand why we only update if $g(x)\le0$.

We need to define a cost function to minimize it, so our cost function will become:
$$
L(w)=\displaystyle\sum\_{x\_i\in U}(-y\_i(w^Tx\_i+b))
$$
where

* $U$is the set of the misclassified points
* $y\_i$ is the label of data point $i$-th
* $x\_i$ is the $i$-th data vector
* $w$ and $b$ is parameters of our model

For each data point, we have the derivative is
$$
\frac{\partial L}{\partial w} = -y\_ix\_i \\
\frac{\partial L}{\partial b} = -y\_i
$$

Finally, update them by Stochastic gradient descent (SGD), we get:
$$
w = w - \frac{\partial L}{\partial w} = w + y\_ix\_i \\
b = b - \frac{\partial L}{\partial b} = b + y\_i
$$

For your last question, notice that the weight and bias changed from $4$-th updated, so we have:
$$
y\_5 = 1, x\_5 = (4,8), w = (-2,-2), b = -2 \\
\Rightarrow g\_5 = +1 \times (4\times(-2) + 8\times(-2)+ (-2)) = -8 -16 -2
$$


---

