Answers.Body: <p>Imagine we have the curve <span class="math-container">$f(x) = x^2$</span>, and we want to find the minimum of this function. The derivate of <span class="math-container">$f$</span> with respect to <span class="math-container">$x$</span> is <span class="math-container">$2x$</span>. Now, gradient descent works by updating our current estimate of the minimum, say <span class="math-container">$c_t$</span>, by the following iterative process
  <span class="math-container">$$c_{t+1} = c_t - \alpha \times \nabla_xf(x=c_t),$$</span>
  where <span class="math-container">$\alpha$</span> is some constant to control how much we shift towards the gradient.</p>
  <p>Intuitively, this should make sense. Imagine our current estimate of the minimum is <span class="math-container">$c_t = -1$</span>. The update would then give us <span class="math-container">$c_{t+1} = -1 - \alpha \times -2 = -1 + 2\alpha &gt; -1$</span>. As you can see, the update has shifted our estimate in the direction of the minimum. If our estimate were <span class="math-container">$+1$</span> then you can probably see that the update would again have shifted us in the direction of the minimum again.</p>
  <p>Now, what happens in machine learning is we have a loss function <span class="math-container">$L$</span> that we typically want to find the minimum of in terms of the parameters of our model. By applying gradient descent on the loss function as we did above with <span class="math-container">$f(x)$</span> we iteratively apply the update rule which will eventually lead us to the minimum of our loss function with respect to the weights. The process is exactly the same as above except it is likely to happen in higher dimensions, where the derivative of <span class="math-container">$f$</span> becomes a vector of partial derivates. Note that whilst <span class="math-container">$w_i$</span>'s are scalar values, we are not differentiating these values, rather we are differentiating the loss function with respect to these scalars.</p>
  <p>I would recommend you to try this out with a simple linear model of maybe 2 or 3 parameters. <a href="https://towardsdatascience.com/linear-regression-using-gradient-descent-97a6c8700931" rel="nofollow noreferrer">After a quick Google I found this article that may be useful</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-24T22:59:33.263Z
Answers.LastActivityDate: 2020-09-24T22:59:33.263Z
Answers.UserDisplayName: David
Body: <p>I am trying to comprehend how the Gradient Descent works.</p>
  <p>I understand we have a cost function which is defined in terms of the following parameters,</p>
  <p><span class="math-container">$J(_{1},_{2},.... , w_{n}, b)$</span></p>
  <p>the derivative would tell us which direction to adjust the parameters.</p>
  <p>i.e. <span class="math-container">$\dfrac{dJ(_{1},_{2},.... , w_{n}, b)}{d(_{1}}$</span> is the rate of change of the cost w.r.t <span class="math-container">$$</span></p>
  <p>The lecture kept on saying this very valuable as we are asking the question <em>how should I change <span class="math-container">$$</span></em> to improve the cost?</p>
  <p>But then the Lecturer presented <span class="math-container">$w_{1}$</span>, <span class="math-container">$w_{2}$</span>, ... as scaler value, How can we differentiate a scalar value.</p>
  <p>I am fundamentally missing what is happening.</p>
  <p>Can anyone please guide me to any blog post, a book that I should read to understand better?</p>
  
Tags: <backpropagation><gradient-descent><linear-regression>
Title: How parameter adjustment works in Gradient Descent?
UserDisplayName: Exploring
ThreadText: # How parameter adjustment works in Gradient Descent?
  
  (Asked by: username_0 on 2020-09-24)
  
  I am trying to comprehend how the Gradient Descent works.
  
  I understand we have a cost function which is defined in terms of the following parameters,
  
  $J(\_{1},\_{2},.... , w\_{n}, b)$
  
  the derivative would tell us which direction to adjust the parameters.
  
  i.e. $\dfrac{dJ(\_{1},\_{2},.... , w\_{n}, b)}{d(\_{1}}$ is the rate of change of the cost w.r.t $$
  
  The lecture kept on saying this very valuable as we are asking the question *how should I change $$* to improve the cost?
  
  But then the Lecturer presented $w\_{1}$, $w\_{2}$, ... as scaler value, How can we differentiate a scalar value.
  
  I am fundamentally missing what is happening.
  
  Can anyone please guide me to any blog post, a book that I should read to understand better?
  
  **Tags:** backpropagation, gradient-descent, linear-regression
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-24. Score: 3
  
  Imagine we have the curve $f(x) = x^2$, and we want to find the minimum of this function. The derivate of $f$ with respect to $x$ is $2x$. Now, gradient descent works by updating our current estimate of the minimum, say $c\_t$, by the following iterative process
  $$c\_{t+1} = c\_t - \alpha \times \nabla\_xf(x=c\_t),$$
  where $\alpha$ is some constant to control how much we shift towards the gradient.
  
  Intuitively, this should make sense. Imagine our current estimate of the minimum is $c\_t = -1$. The update would then give us $c\_{t+1} = -1 - \alpha \times -2 = -1 + 2\alpha > -1$. As you can see, the update has shifted our estimate in the direction of the minimum. If our estimate were $+1$ then you can probably see that the update would again have shifted us in the direction of the minimum again.
  
  Now, what happens in machine learning is we have a loss function $L$ that we typically want to find the minimum of in terms of the parameters of our model. By applying gradient descent on the loss function as we did above with $f(x)$ we iteratively apply the update rule which will eventually lead us to the minimum of our loss function with respect to the weights. The process is exactly the same as above except it is likely to happen in higher dimensions, where the derivative of $f$ becomes a vector of partial derivates. Note that whilst $w\_i$'s are scalar values, we are not differentiating these values, rather we are differentiating the loss function with respect to these scalars.
  
  I would recommend you to try this out with a simple linear model of maybe 2 or 3 parameters. [After a quick Google I found this article that may be useful](https://towardsdatascience.com/linear-regression-using-gradient-descent-97a6c8700931).
  
  
  ---
  
  