Answers.Body: <p><span class="math-container">$$let,\ x_{(n)} \ be\ a\ point\ on\ x-axis\ where\ f'( x) \ =\ 0\ ,\ and\ x_{(n\ +\ h)} \ is\ any\ other\ arbitary\ point \\
  \therefore \ \ \frac{f'( x_{(n\ +\ h)})}{|\ f'( x_{(n\ +\ h)}) \ |} =\begin{cases}
  1 &amp; \mathrm{if,} \ h\ \  &gt;0\\
  0 &amp; \mathrm{if,} \ h\ =\ 0\\
  -1 &amp; \mathrm{if,} \ h\ &lt; \ 0
  \end{cases}\\similarly,\ \ \frac{x_{(n)} \ -\ x_{(n\ +\ h)} \ }{|\ x_{(n)} \ -\ x_{(n\ +\ h)} \ |} \ =\ \begin{cases}
  1 &amp; \mathrm{if,} \ h\ \ &lt; 0\\
  0 &amp; \mathrm{if,} \ h\ =\ 0\\
  -1 &amp; \mathrm{if,} \ h\  &gt;\ 0
  \end{cases}\\or,\ \frac{x_{(n)} \ -\ x_{(n\ +\ h)} \ }{|\ x_{(n)} \ -\ x_{(n\ +\ h)} \ |} \ =\ -\ \ \ \frac{f'( x_{(n\ +\ h)})}{|\ f'( x_{(n\ +\ h)}) \ |}\\ 
  \therefore \ x_{(n)} \ =x_{(n\ +\ h)} \ -\ \eta \times f'( x_{(n\ +\ h)}) \ \ \ \ \ \ \left[ where\ \eta \ =\frac{|\ x_{(n)} \ -\ x_{(n\ +\ h)} \ |}{|f'( x_{(n\ +\ h)}) \ |} \ \right]$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-01-21T16:40:54.327Z
Answers.LastActivityDate: 2020-01-25T13:21:52.293Z
Answers.LastEditDate: 2020-01-25T13:21:52.293Z
Answers.UserDisplayName: souparno majumder
Answers.Body: <p>Don’t think about it as the <span class="math-container">$w_{(n)}-w_{(n+1)}$</span> being proportional to something. Think about it this way:</p>
  <p>I'm now at <span class="math-container">$w_{(n)}$</span>. Where do I want to be at timestep, so that the error decreases? For that, I need to know how the error changes when I make small steps to the left or right of <span class="math-container">$w_{(n)}$</span>.</p>
  <p>If <span class="math-container">$E$</span> increases as I increase <span class="math-container">$w$</span> (that is, if <span class="math-container">$\frac{\partial E}{\partial w}&gt;0$</span>, then obviously, I would want to move a little bit to the left. In other words, <span class="math-container">$w_{n+1}&lt;w_{n}$</span> or <span class="math-container">$w_{n+1}-w_{n}&lt;0$</span>.</p>
  <p>On the other hand, if the derivative were negative, you know that you should move right to reduce the error a little bit, <span class="math-container">$w_{n+1}-w_{n}&gt;0$</span>.  So, basically, your step should have the opposite sign of the derivative.</p>
  <p><span class="math-container">$$w_{n+1}-w_{n} \propto-\frac{\partial E}{\partial w}$$</span></p>
  <p><span class="math-container">$\alpha$</span>, the learning rate, is just the constant of proportionality.
  Caution: think about small values for this rate, not big numbers.  Taking a huge step can cause you to overshoot the minimum point.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-01-21T11:41:15.567Z
Answers.LastActivityDate: 2020-07-06T23:13:14.173Z
Answers.LastEditDate: 2020-07-06T23:13:14.173Z
Answers.UserDisplayName: Mathemagical
Body: <p>In the [delta rule][1] the equation to adjust the weight with respect to error is</p>
  <p><span class="math-container">$$w_{(n+1)}=w_{(n)}-\alpha \times \frac{\partial E}{\partial w}$$</span></p>
  <p>*where <span class="math-container">$\alpha$</span> is the learning rate and <span class="math-container">$E$</span> is the error.</p>
  <p>The graph for <span class="math-container">$E$</span> vs <span class="math-container">$w$</span> would look like the one below with <span class="math-container">$E$</span> in the <span class="math-container">$y$</span> axis and <span class="math-container">$W$</span> in the <span class="math-container">$x$</span>-axis</p>
  <p><a href="https://i.sstatic.net/Ju05S.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Ju05S.png" alt="enter image description here" /></a></p>
  <p>In other words, we can write</p>
  <p><span class="math-container">$$\alpha \times \frac{\partial E}{\partial w}=w_{(n)}-w_{(n+1)}$$</span></p>
  <p>I want to know, what is the proof behind the gradient of a curve being equal/proportional to the distance between the two coordinates in the x-axis.</p>
  <p><span class="math-container">$\frac{\partial E}{\partial w}$</span> times step is a small shift on <span class="math-container">$f(w)$</span> not <span class="math-container">$w$</span>. So, why does the difference between <span class="math-container">$W(n+1)$</span> and <span class="math-container">$W(n)$</span> be equal to <span class="math-container">$f(W)$</span>?</p>
  <p>I found <a href="https://stats.stackexchange.com/q/305516/82135">a similar question</a>, but the accepted answer doesn't have a proof.</p>
  
Tags: <machine-learning><backpropagation><learning-algorithms><gradient-descent>
Title: What is the proof behind the gradient of a curve being proportional to the distance between the two co-ordinates in the x-axis?
UserDisplayName: souparno majumder
ThreadText: # What is the proof behind the gradient of a curve being proportional to the distance between the two co-ordinates in the x-axis?
  
  (Asked by: username_0 on 2018-01-20)
  
  In the [delta rule][1] the equation to adjust the weight with respect to error is
  
  $$w\_{(n+1)}=w\_{(n)}-\alpha \times \frac{\partial E}{\partial w}$$
  
  \*where $\alpha$ is the learning rate and $E$ is the error.
  
  The graph for $E$ vs $w$ would look like the one below with $E$ in the $y$ axis and $W$ in the $x$-axis
  
  [![enter image description here](https://i.sstatic.net/Ju05S.png)](https://i.sstatic.net/Ju05S.png)
  
  In other words, we can write
  
  $$\alpha \times \frac{\partial E}{\partial w}=w\_{(n)}-w\_{(n+1)}$$
  
  I want to know, what is the proof behind the gradient of a curve being equal/proportional to the distance between the two coordinates in the x-axis.
  
  $\frac{\partial E}{\partial w}$ times step is a small shift on $f(w)$ not $w$. So, why does the difference between $W(n+1)$ and $W(n)$ be equal to $f(W)$?
  
  I found [a similar question](https://stats.stackexchange.com/q/305516/82135), but the accepted answer doesn't have a proof.
  
  **Tags:** machine-learning, backpropagation, learning-algorithms, gradient-descent
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-01-21. Score: 2 | Accepted answer
  
  Don’t think about it as the $w\_{(n)}-w\_{(n+1)}$ being proportional to something. Think about it this way:
  
  I'm now at $w\_{(n)}$. Where do I want to be at timestep, so that the error decreases? For that, I need to know how the error changes when I make small steps to the left or right of $w\_{(n)}$.
  
  If $E$ increases as I increase $w$ (that is, if $\frac{\partial E}{\partial w}>0$, then obviously, I would want to move a little bit to the left. In other words, $w\_{n+1} or $w\_{n+1}-w\_{n}<0$.
  
  On the other hand, if the derivative were negative, you know that you should move right to reduce the error a little bit, $w\_{n+1}-w\_{n}>0$. So, basically, your step should have the opposite sign of the derivative.
  
  $$w\_{n+1}-w\_{n} \propto-\frac{\partial E}{\partial w}$$
  
  $\alpha$, the learning rate, is just the constant of proportionality.
  Caution: think about small values for this rate, not big numbers. Taking a huge step can cause you to overshoot the minimum point.
  
  
  ---
  
  ### Answer by username_0 on 2018-01-21. Score: 0
  
  $$let,\ x\_{(n)} \ be\ a\ point\ on\ x-axis\ where\ f'( x) \ =\ 0\ ,\ and\ x\_{(n\ +\ h)} \ is\ any\ other\ arbitary\ point \\
  \therefore \ \ \frac{f'( x\_{(n\ +\ h)})}{|\ f'( x\_{(n\ +\ h)}) \ |} =\begin{cases}
  1 & \mathrm{if,} \ h\ \ >0\\
  0 & \mathrm{if,} \ h\ =\ 0\\
  -1 & \mathrm{if,} \ h\ < \ 0
  \end{cases}\\similarly,\ \ \frac{x\_{(n)} \ -\ x\_{(n\ +\ h)} \ }{|\ x\_{(n)} \ -\ x\_{(n\ +\ h)} \ |} \ =\ \begin{cases}
  1 & \mathrm{if,} \ h\ \ < 0\\
  0 & \mathrm{if,} \ h\ =\ 0\\
  -1 & \mathrm{if,} \ h\ >\ 0
  \end{cases}\\or,\ \frac{x\_{(n)} \ -\ x\_{(n\ +\ h)} \ }{|\ x\_{(n)} \ -\ x\_{(n\ +\ h)} \ |} \ =\ -\ \ \ \frac{f'( x\_{(n\ +\ h)})}{|\ f'( x\_{(n\ +\ h)}) \ |}\\
  \therefore \ x\_{(n)} \ =x\_{(n\ +\ h)} \ -\ \eta \times f'( x\_{(n\ +\ h)}) \ \ \ \ \ \ \left[ where\ \eta \ =\frac{|\ x\_{(n)} \ -\ x\_{(n\ +\ h)} \ |}{|f'( x\_{(n\ +\ h)}) \ |} \ \right]$$
  
  
  ---
  
  