Answers.Body: <p>The method you propose is already known, its basically a numerical approximation to the gradient. It is not used to train neural networks because its well... an approximation. You still need to do two forward passes to get an approximation, which introduces noise and might make the training process fail.</p>
  
  <p>Using backpropagation to compute the gradient is an exact solution, so why would you use an approximation if the exact computation is equally efficient?</p>
  
  <p>Numeric approximations of the gradient only make sense if exact computation is not possible.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-19T00:09:59.650Z
Answers.Text: Isn't it much more computationally intensive to compute the gradient every single time in contrast to two forward propagation?
Answers.UserDisplayName: Aphrodite
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-19T00:13:55.817Z
Answers.Text: @Aphrodite No, the gradient computed by backpropagation is very efficient, and exact, your approximation is not really faster.
Answers.UserDisplayName: Dr. Snoopy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-18T18:11:52.880Z
Answers.LastActivityDate: 2020-01-18T18:11:52.880Z
Answers.UserDisplayName: Dr. Snoopy
Body: <p>I am reading about backpropagation and I wonder why I have to backpropagate. </p>
  
  <p>For example, I would update the network by randomly choosing a weight to change, <span class="math-container">$w$</span>. I would have <span class="math-container">$X$</span> and <span class="math-container">$y$</span>. Then, I would choose <span class="math-container">$dw$</span>, a random number from <span class="math-container">$-0.1$</span> to <span class="math-container">$0.1$</span>, for example. Then, I would do two predictions of the neural network and get their losses with the original neural network and one with <span class="math-container">$w$</span> changed by <span class="math-container">$dw$</span> to get the respective losses <span class="math-container">$L_{\text{original}}$</span> and <span class="math-container">$L_{\text{updated}}$</span>. <span class="math-container">$L_{\text{updated}} - L_{\text{original}}$</span> is <span class="math-container">$dL$</span>. I would update <span class="math-container">$w$</span> by <span class="math-container">$\gamma \frac{d L}{dw}$</span>, where <span class="math-container">$\gamma$</span> is the learning rate and <span class="math-container">$L$</span> is the loss.</p>
  
  <p>This does not need a gradient backpropagation throughout the system, and must have somehow a disadvantage because no one uses it. What is this disadvantage?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-18T18:23:50.877Z
Comments.Text: Since there's already a good answer I'd like to point out that BP only requires one forward pass (per update) whereas this method requires 2.
Comments.UserDisplayName: respectful
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-24T06:42:37.417Z
Comments.Text: You might find this interesting/related: https://arxiv.org/abs/1707.04585 
  
  Reversible residual layers are another approach at optimizing backpropagation. Google recently announced Reformer, a transformer model that somewhat leverages this: https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html
Comments.UserDisplayName: oeste
Tags: <neural-networks><deep-learning><backpropagation>
Title: How is back-propagation useful in neural networks?
UserDisplayName: Aphrodite
ThreadText: # How is back-propagation useful in neural networks?
  
  (Asked by: username_0 on 2020-01-18)
  
  I am reading about backpropagation and I wonder why I have to backpropagate.
  
  For example, I would update the network by randomly choosing a weight to change, $w$. I would have $X$ and $y$. Then, I would choose $dw$, a random number from $-0.1$ to $0.1$, for example. Then, I would do two predictions of the neural network and get their losses with the original neural network and one with $w$ changed by $dw$ to get the respective losses $L\_{\text{original}}$ and $L\_{\text{updated}}$. $L\_{\text{updated}} - L\_{\text{original}}$ is $dL$. I would update $w$ by $\gamma \frac{d L}{dw}$, where $\gamma$ is the learning rate and $L$ is the loss.
  
  This does not need a gradient backpropagation throughout the system, and must have somehow a disadvantage because no one uses it. What is this disadvantage?
  
  **Tags:** neural-networks, deep-learning, backpropagation
  
  > **Comments:**
  > **username_1**: Since there's already a good answer I'd like to point out that BP only requires one forward pass (per update) whereas this method requires 2.  
  > **username_2**: You might find this interesting/related: https://arxiv.org/abs/1707.04585   Reversible residual layers are another approach at optimizing backpropagation. Google recently announced Reformer, a transformer model that somewhat leverages this: https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2020-01-18. Score: 3 | Accepted answer
  
  The method you propose is already known, its basically a numerical approximation to the gradient. It is not used to train neural networks because its well... an approximation. You still need to do two forward passes to get an approximation, which introduces noise and might make the training process fail.
  
  Using backpropagation to compute the gradient is an exact solution, so why would you use an approximation if the exact computation is equally efficient?
  
  Numeric approximations of the gradient only make sense if exact computation is not possible.
  
  > **Comments:**
  > **username_0**: Isn't it much more computationally intensive to compute the gradient every single time in contrast to two forward propagation?  
  > **username_3**: @username_0 No, the gradient computed by backpropagation is very efficient, and exact, your approximation is not really faster.  
  
  ---
  
  