<p><span class="math-container">$$let,\ x_{(n)} \ be\ a\ point\ on\ x-axis\ where\ f'( x) \ =\ 0\ ,\ and\ x_{(n\ +\ h)} \ is\ any\ other\ arbitary\ point \\
\therefore \ \ \frac{f'( x_{(n\ +\ h)})}{|\ f'( x_{(n\ +\ h)}) \ |} =\begin{cases}
1 &amp; \mathrm{if,} \ h\ \  &gt;0\\
0 &amp; \mathrm{if,} \ h\ =\ 0\\
-1 &amp; \mathrm{if,} \ h\ &lt; \ 0
\end{cases}\\similarly,\ \ \frac{x_{(n)} \ -\ x_{(n\ +\ h)} \ }{|\ x_{(n)} \ -\ x_{(n\ +\ h)} \ |} \ =\ \begin{cases}
1 &amp; \mathrm{if,} \ h\ \ &lt; 0\\
0 &amp; \mathrm{if,} \ h\ =\ 0\\
-1 &amp; \mathrm{if,} \ h\  &gt;\ 0
\end{cases}\\or,\ \frac{x_{(n)} \ -\ x_{(n\ +\ h)} \ }{|\ x_{(n)} \ -\ x_{(n\ +\ h)} \ |} \ =\ -\ \ \ \frac{f'( x_{(n\ +\ h)})}{|\ f'( x_{(n\ +\ h)}) \ |}\\ 
\therefore \ x_{(n)} \ =x_{(n\ +\ h)} \ -\ \eta \times f'( x_{(n\ +\ h)}) \ \ \ \ \ \ \left[ where\ \eta \ =\frac{|\ x_{(n)} \ -\ x_{(n\ +\ h)} \ |}{|f'( x_{(n\ +\ h)}) \ |} \ \right]$$</span></p>

CC BY-SA 4.0
2018-01-21T16:40:54.327Z
2020-01-25T13:21:52.293Z
2020-01-25T13:21:52.293Z
souparno majumder
<p>Don’t think about it as the <span class="math-container">$w_{(n)}-w_{(n+1)}$</span> being proportional to something. Think about it this way:</p>
<p>I'm now at <span class="math-container">$w_{(n)}$</span>. Where do I want to be at timestep, so that the error decreases? For that, I need to know how the error changes when I make small steps to the left or right of <span class="math-container">$w_{(n)}$</span>.</p>
<p>If <span class="math-container">$E$</span> increases as I increase <span class="math-container">$w$</span> (that is, if <span class="math-container">$\frac{\partial E}{\partial w}&gt;0$</span>, then obviously, I would want to move a little bit to the left. In other words, <span class="math-container">$w_{n+1}&lt;w_{n}$</span> or <span class="math-container">$w_{n+1}-w_{n}&lt;0$</span>.</p>
<p>On the other hand, if the derivative were negative, you know that you should move right to reduce the error a little bit, <span class="math-container">$w_{n+1}-w_{n}&gt;0$</span>.  So, basically, your step should have the opposite sign of the derivative.</p>
<p><span class="math-container">$$w_{n+1}-w_{n} \propto-\frac{\partial E}{\partial w}$$</span></p>
<p><span class="math-container">$\alpha$</span>, the learning rate, is just the constant of proportionality.
Caution: think about small values for this rate, not big numbers.  Taking a huge step can cause you to overshoot the minimum point.</p>

CC BY-SA 4.0
2018-01-21T11:41:15.567Z
2020-07-06T23:13:14.173Z
2020-07-06T23:13:14.173Z
Mathemagical
<p>In the [delta rule][1] the equation to adjust the weight with respect to error is</p>
<p><span class="math-container">$$w_{(n+1)}=w_{(n)}-\alpha \times \frac{\partial E}{\partial w}$$</span></p>
<p>*where <span class="math-container">$\alpha$</span> is the learning rate and <span class="math-container">$E$</span> is the error.</p>
<p>The graph for <span class="math-container">$E$</span> vs <span class="math-container">$w$</span> would look like the one below with <span class="math-container">$E$</span> in the <span class="math-container">$y$</span> axis and <span class="math-container">$W$</span> in the <span class="math-container">$x$</span>-axis</p>
<p><a href="https://i.sstatic.net/Ju05S.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Ju05S.png" alt="enter image description here" /></a></p>
<p>In other words, we can write</p>
<p><span class="math-container">$$\alpha \times \frac{\partial E}{\partial w}=w_{(n)}-w_{(n+1)}$$</span></p>
<p>I want to know, what is the proof behind the gradient of a curve being equal/proportional to the distance between the two coordinates in the x-axis.</p>
<p><span class="math-container">$\frac{\partial E}{\partial w}$</span> times step is a small shift on <span class="math-container">$f(w)$</span> not <span class="math-container">$w$</span>. So, why does the difference between <span class="math-container">$W(n+1)$</span> and <span class="math-container">$W(n)$</span> be equal to <span class="math-container">$f(W)$</span>?</p>
<p>I found <a href="https://stats.stackexchange.com/q/305516/82135">a similar question</a>, but the accepted answer doesn't have a proof.</p>

<machine-learning><backpropagation><learning-algorithms><gradient-descent>
What is the proof behind the gradient of a curve being proportional to the distance between the two co-ordinates in the x-axis?
souparno majumder
# What is the proof behind the gradient of a curve being proportional to the distance between the two co-ordinates in the x-axis?

(Asked by: username_0 on 2018-01-20)

In the [delta rule][1] the equation to adjust the weight with respect to error is

$$w\_{(n+1)}=w\_{(n)}-\alpha \times \frac{\partial E}{\partial w}$$

\*where $\alpha$ is the learning rate and $E$ is the error.

The graph for $E$ vs $w$ would look like the one below with $E$ in the $y$ axis and $W$ in the $x$-axis

[![enter image description here](https://i.sstatic.net/Ju05S.png)](https://i.sstatic.net/Ju05S.png)

In other words, we can write

$$\alpha \times \frac{\partial E}{\partial w}=w\_{(n)}-w\_{(n+1)}$$

I want to know, what is the proof behind the gradient of a curve being equal/proportional to the distance between the two coordinates in the x-axis.

$\frac{\partial E}{\partial w}$ times step is a small shift on $f(w)$ not $w$. So, why does the difference between $W(n+1)$ and $W(n)$ be equal to $f(W)$?

I found [a similar question](https://stats.stackexchange.com/q/305516/82135), but the accepted answer doesn't have a proof.

**Tags:** machine-learning, backpropagation, learning-algorithms, gradient-descent


---

## 2 answers

### Answer by username_1 on 2018-01-21. Score: 2 | Accepted answer

Don’t think about it as the $w\_{(n)}-w\_{(n+1)}$ being proportional to something. Think about it this way:

I'm now at $w\_{(n)}$. Where do I want to be at timestep, so that the error decreases? For that, I need to know how the error changes when I make small steps to the left or right of $w\_{(n)}$.

If $E$ increases as I increase $w$ (that is, if $\frac{\partial E}{\partial w}>0$, then obviously, I would want to move a little bit to the left. In other words, $w\_{n+1} or $w\_{n+1}-w\_{n}<0$.

On the other hand, if the derivative were negative, you know that you should move right to reduce the error a little bit, $w\_{n+1}-w\_{n}>0$. So, basically, your step should have the opposite sign of the derivative.

$$w\_{n+1}-w\_{n} \propto-\frac{\partial E}{\partial w}$$

$\alpha$, the learning rate, is just the constant of proportionality.
Caution: think about small values for this rate, not big numbers. Taking a huge step can cause you to overshoot the minimum point.


---

### Answer by username_0 on 2018-01-21. Score: 0

$$let,\ x\_{(n)} \ be\ a\ point\ on\ x-axis\ where\ f'( x) \ =\ 0\ ,\ and\ x\_{(n\ +\ h)} \ is\ any\ other\ arbitary\ point \\
\therefore \ \ \frac{f'( x\_{(n\ +\ h)})}{|\ f'( x\_{(n\ +\ h)}) \ |} =\begin{cases}
1 & \mathrm{if,} \ h\ \ >0\\
0 & \mathrm{if,} \ h\ =\ 0\\
-1 & \mathrm{if,} \ h\ < \ 0
\end{cases}\\similarly,\ \ \frac{x\_{(n)} \ -\ x\_{(n\ +\ h)} \ }{|\ x\_{(n)} \ -\ x\_{(n\ +\ h)} \ |} \ =\ \begin{cases}
1 & \mathrm{if,} \ h\ \ < 0\\
0 & \mathrm{if,} \ h\ =\ 0\\
-1 & \mathrm{if,} \ h\ >\ 0
\end{cases}\\or,\ \frac{x\_{(n)} \ -\ x\_{(n\ +\ h)} \ }{|\ x\_{(n)} \ -\ x\_{(n\ +\ h)} \ |} \ =\ -\ \ \ \frac{f'( x\_{(n\ +\ h)})}{|\ f'( x\_{(n\ +\ h)}) \ |}\\
\therefore \ x\_{(n)} \ =x\_{(n\ +\ h)} \ -\ \eta \times f'( x\_{(n\ +\ h)}) \ \ \ \ \ \ \left[ where\ \eta \ =\frac{|\ x\_{(n)} \ -\ x\_{(n\ +\ h)} \ |}{|f'( x\_{(n\ +\ h)}) \ |} \ \right]$$


---

