Answers.Body: <p>Answer by <a href="https://math.stackexchange.com/users/248286/theo-bendit">Theo Bandit</a> <a href="https://math.stackexchange.com/questions/3160982/how-is-x-1-x-2-normal-to-x-1w-1-x-2w-2-y/3160988#3160988">at maths stackexchange</a></p>
  
  <blockquote>
    <p>If you choose two points <span class="math-container">$(w_1, w_2), (v_1, v_2)$</span> along this line, then
    <span class="math-container">$$(x_1, x_2) \cdot ((w_1, w_2) - (v_1, v_2)) = x_1 w_1 + x_2 w_2 - (x_1 v_1 + x_2 v_2) = y - y = 0.$$</span>
    That is, the direction <span class="math-container">$(x_1, x_2)$</span> is perpendicular to any vector lying along the line, i.e. <span class="math-container">$(x_1, x_2)$</span> is normal to the line.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-24T20:38:03.707Z
Answers.LastActivityDate: 2019-03-24T20:38:03.707Z
Answers.UserDisplayName: EmmanuelMess
Body: <p><a href="https://page.mi.fu-berlin.de/rojas/neural/chapter/K8.pdf#page=9" rel="nofollow noreferrer">Raul Rojas' Neural Networks A Systematic Introduction, section 8.1.2</a> relates off-line backpropagation and on-line backpropagation with Gauss-Jacobi and Gauss-Seidel methods for finding the intersection of two lines.</p>
  
  <p>What I can't understand is how the iterations of on-line backpropagation are perpendicular to the (current) constraint. More specifically, how is <span class="math-container">$\frac12(x_1w_1 + x_2w_2 - y)^2$</span>'s gradient, <span class="math-container">$(x_1,x_2)$</span>, normal to the constraint <span class="math-container">$x_1w_1 + x_2w_2 = y$</span>?</p>
  
Tags: <backpropagation><math><gradient-descent>
Title: Are on-line backpropagation iterations perpendicular to the constraint?
UserDisplayName: EmmanuelMess
ThreadText: # Are on-line backpropagation iterations perpendicular to the constraint?
  
  (Asked by: username_0 on 2019-03-23)
  
  [Raul Rojas' Neural Networks A Systematic Introduction, section 8.1.2](https://page.mi.fu-berlin.de/rojas/neural/chapter/K8.pdf#page=9) relates off-line backpropagation and on-line backpropagation with Gauss-Jacobi and Gauss-Seidel methods for finding the intersection of two lines.
  
  What I can't understand is how the iterations of on-line backpropagation are perpendicular to the (current) constraint. More specifically, how is $\frac12(x\_1w\_1 + x\_2w\_2 - y)^2$'s gradient, $(x\_1,x\_2)$, normal to the constraint $x\_1w\_1 + x\_2w\_2 = y$?
  
  **Tags:** backpropagation, math, gradient-descent
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-03-24. Score: 1 | Accepted answer
  
  Answer by [Theo Bandit](https://math.stackexchange.com/users/248286/theo-bendit) [at maths stackexchange](https://math.stackexchange.com/questions/3160982/how-is-x-1-x-2-normal-to-x-1w-1-x-2w-2-y/3160988#3160988)
  
  > If you choose two points $(w\_1, w\_2), (v\_1, v\_2)$ along this line, then
  > $$(x\_1, x\_2) \cdot ((w\_1, w\_2) - (v\_1, v\_2)) = x\_1 w\_1 + x\_2 w\_2 - (x\_1 v\_1 + x\_2 v\_2) = y - y = 0.$$
  > That is, the direction $(x\_1, x\_2)$ is perpendicular to any vector lying along the line, i.e. $(x\_1, x\_2)$ is normal to the line.
  
  
  ---
  
  