Answers.Body: <p>I'm pretty sure they're the same thing. Both backpropagation and gradient descent's essential ideas are to calculate the partial derivative of the cost with respect to each weight, and subtract the partial derivative times the learning rate.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-01T03:30:34.793Z
Answers.Text: Yes, I believe both are the same. They are different methods and backpropagation is a faster algorithm. This is mentioned and explained why in [here] (http://neuralnetworksanddeeplearning.com/chap2.html#in_what_sense_is_backpropagation_a_fast_algorithm)
Answers.UserDisplayName: Hrushi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-31T15:44:32.933Z
Answers.LastActivityDate: 2020-03-31T15:44:32.933Z
Answers.UserDisplayName: doggosvsshibes
Body: <p>We require to find the gradient of loss function(cost function) w.r.t to the weights to use optimization methods such as SGD or gradient descent. So far, I have come across two ways to compute the gradient:</p>
  
  <ol>
  <li>BackPropagation</li>
  <li>Calculating gradient of loss function by calculus</li>
  </ol>
  
  <p>I found many resources for understanding backpropagation. 
  The 2nd method I am referring to is the image below(taken for a specific example, e is the error: difference between target and prediction):
  <a href="https://i.sstatic.net/OSOVB.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/OSOVB.jpg" alt="enter image description here"></a> </p>
  
  <p>Also, the proof was mentioned in this paper:<a href="https://arxiv.org/pdf/1802.01528.pdf" rel="nofollow noreferrer">here</a></p>
  
  <p>Moreover, I found this method while reading this <a href="https://www.pyimagesearch.com/2016/10/10/gradient-descent-with-python/" rel="nofollow noreferrer">blog</a>.(You might have to scroll down to see the code: gradient = X.T.dot(error) / X.shape[0]  ) </p>
  
  <p>My question is are the two methods of finding gradient of cost function same? It appears different and if yes, which one is more efficient( though one can guess it is backpropagation)</p>
  
  <p>Would be grateful for any help. Thanks for being patient(it's my 1st time learning ML). </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-31T13:41:36.410Z
Comments.Text: But sir, if both are same, why do we use backPropagation, isn't it very lengthy
Comments.UserDisplayName: Hrushi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-04T18:09:10.107Z
Comments.Text: Backpropagation is a method/process of propagating the loss back to previous layers, whereas gradient descent (and it's variants like SGD,RMSprop etc) is a method to optimize the neural network to reduce the loss. Loss is less when we "descent" to a valley point, which we calculate through gradients or slope (ie. derivatives) and we send this information back using backprop where different channels receive different value of the gradient depending on their contribution. We can also send info other than gradients using backprop, there are also methods which don't use one or other, or either
Comments.UserDisplayName: SajanGohil
Tags: <backpropagation><gradient-descent>
Title: Different methods of calculating gradients of cost function(loss function)
UserDisplayName: Hrushi
ThreadText: # Different methods of calculating gradients of cost function(loss function)
  
  (Asked by: username_0 on 2020-03-31)
  
  We require to find the gradient of loss function(cost function) w.r.t to the weights to use optimization methods such as SGD or gradient descent. So far, I have come across two ways to compute the gradient:
  
  1. BackPropagation
  2. Calculating gradient of loss function by calculus
  
  I found many resources for understanding backpropagation.
  The 2nd method I am referring to is the image below(taken for a specific example, e is the error: difference between target and prediction):
  [![enter image description here](https://i.sstatic.net/OSOVB.jpg)](https://i.sstatic.net/OSOVB.jpg)
  
  Also, the proof was mentioned in this paper:[here](https://arxiv.org/pdf/1802.01528.pdf)
  
  Moreover, I found this method while reading this [blog](https://www.pyimagesearch.com/2016/10/10/gradient-descent-with-python/).(You might have to scroll down to see the code: gradient = X.T.dot(error) / X.shape[0] )
  
  My question is are the two methods of finding gradient of cost function same? It appears different and if yes, which one is more efficient( though one can guess it is backpropagation)
  
  Would be grateful for any help. Thanks for being patient(it's my 1st time learning ML).
  
  **Tags:** backpropagation, gradient-descent
  
  > **Comments:**
  > **username_0**: But sir, if both are same, why do we use backPropagation, isn't it very lengthy  
  > **username_1**: Backpropagation is a method/process of propagating the loss back to previous layers, whereas gradient descent (and it's variants like SGD,RMSprop etc) is a method to optimize the neural network to reduce the loss. Loss is less when we "descent" to a valley point, which we calculate through gradients or slope (ie. derivatives) and we send this information back using backprop where different channels receive different value of the gradient depending on their contribution. We can also send info other than gradients using backprop, there are also methods which don't use one or other, or either  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-03-31. Score: 1 | Accepted answer
  
  I'm pretty sure they're the same thing. Both backpropagation and gradient descent's essential ideas are to calculate the partial derivative of the cost with respect to each weight, and subtract the partial derivative times the learning rate.
  
  > **Comments:**
  > **username_0**: Yes, I believe both are the same. They are different methods and backpropagation is a faster algorithm. This is mentioned and explained why in [here] (http://neuralnetworksanddeeplearning.com/chap2.html#in_what_sense_is_backpropagation_a_fast_algorithm)  
  
  ---
  
  