Answers.Body: <p>In your case, <span class="math-container">$L$</span> is the loss (or cost) function, which can be, for example, the <a href="https://en.wikipedia.org/wiki/Mean_squared_error" rel="nofollow noreferrer">mean squared error (MSE)</a> or the cross-entropy, depending on the problem you want to solve. Given one training example <span class="math-container">$(\mathbf{x}_i, y_i) \in D$</span>, where <span class="math-container">$\mathbf{x}_i \in \mathbb{R}^d$</span> is the input (for example, an image) and <span class="math-container">$y_i \in \mathbb{R}$</span> can either be a label (aka class) or a numerical value, and <span class="math-container">$D$</span> is your training dataset, then the <a href="https://en.wikipedia.org/wiki/Mean_squared_error" rel="nofollow noreferrer">MSE</a> is defined as follows</p>
  
  <p><span class="math-container">$$L(\mathbf{W}) = \frac{1}{2} \left(f(\mathbf{x}_i) - y_i \right)^2,$$</span></p>
  
  <p>where <span class="math-container">$f(\mathbf{x}_i) \in \mathbb{R}$</span> is the output of the neural network <span class="math-container">$f$</span> given the input <span class="math-container">$\mathbf{x}_i$</span>.</p>
  
  <p>If you have a mini-batch of <span class="math-container">$M$</span> training examples <span class="math-container">$\{(\mathbf{x}_i, y_i) \}_{i=1}^M$</span>, then the loss will be an average of the MSE for each training example. For more info, have a look at this answer <a href="https://ai.stackexchange.com/a/11675/2444">https://ai.stackexchange.com/a/11675/2444</a>. The <a href="https://ai.stackexchange.com/a/8985/2444">https://ai.stackexchange.com/a/8985/2444</a> may also be useful.</p>
  
  <p>See the article <a href="https://machinelearningmastery.com/loss-and-loss-functions-for-training-deep-learning-neural-networks/" rel="nofollow noreferrer">Loss and Loss Functions for Training Deep Learning Neural Networks</a> for more info regarding different losses used in deep learning and how to choose the appropriate loss for your problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-30T05:54:06.823Z
Answers.Text: Is that true to say : L(w) = 1/2 * (f(x,w) - y)**2 ?
Answers.UserDisplayName: Mahdi Amrollahi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-30T06:50:26.960Z
Answers.Text: I did not get my answer completely however, your answer was very helpful.
  You said in this answer: (
  https://ai.stackexchange.com/questions/11667/is-back-propagation-applied-for-each-data-point-or-for-a-batch-of-data-points/11675#11675 )
  "For simplicity, assume that we are able to calculate the gradient of L, that is, delat(L)". So, how can I calculate the gradient of L function according to W set, because I need to update my W set. I mean that, do we have such a function like f(x) = 2x and then calculate the L(w)=1/2*(2x-y)**2 and delta(L) = 2*(2x-y)?
Answers.UserDisplayName: Mahdi Amrollahi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-30T12:12:14.303Z
Answers.Text: $f(x, w)$ or $f(x)$ are "just" notations, in the sense that $w$ is clearly a parameter (or weight) of the neural network $x$, but $w$ is not an input to the neural network: $x$ is the input! The way you calculate the gradient of $L$ with respect to any of the parameters is with **back-propagation** and it depends on the architecture of the neural network and the loss function you choose. To understand back-propagation, you need to understand the basics of calculus.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-30T13:53:07.040Z
Answers.Text: Thanks and I got the point :
  https://towardsdatascience.com/linear-regression-using-gradient-descent-97a6c8700931
Answers.UserDisplayName: Mahdi Amrollahi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-29T18:52:02.317Z
Answers.LastActivityDate: 2019-12-29T18:57:12.543Z
Answers.LastEditDate: 2019-12-29T18:57:12.543Z
Answers.UserDisplayName: nbro
Body: <p>The gradient descent step is the following</p>
  
  <p><span class="math-container">\begin{align}
  \mathbf{W}_i = \mathbf{W}_{i-1} - \alpha * \nabla L(\mathbf{W}_{i-1})
  \end{align}</span></p>
  
  <p>were <span class="math-container">$L(\mathbf{W}_{i-1})$</span> is the loss value, <span class="math-container">$\alpha$</span> the learning rate and <span class="math-container">$\nabla L(\mathbf{W}_{i-1})$</span> the gradient of the loss.</p>
  
  <p>So, how do we get to the <span class="math-container">$L(\mathbf{W}_{i-1})$</span> to calculate the gradient of <span class="math-container">$L(\mathbf{W}_{i-1})$</span>? As an example, we can initialize the set of <span class="math-container">$\mathbf{W}$</span> to 0.5. How can you explain it to me?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-29T18:39:14.070Z
Comments.Text: Are you asking how to calculate the gradient of $f$ or how to calculate the loss value itself?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-29T18:43:43.137Z
Comments.Text: Wow, Thanks for your edition. Before I can to calculate the gradient of f(W), I should have the f function. So, how can I have the f function at the first?
Comments.UserDisplayName: Mahdi Amrollahi
Tags: <deep-learning><gradient-descent>
Title: How is the loss value calculated in order to compute the gradient?
UserDisplayName: Mahdi Amrollahi
ThreadText: # How is the loss value calculated in order to compute the gradient?
  
  (Asked by: username_0 on 2019-12-29)
  
  The gradient descent step is the following
  
  \begin{align}
  \mathbf{W}\_i = \mathbf{W}\_{i-1} - \alpha \* \nabla L(\mathbf{W}\_{i-1})
  \end{align}
  
  were $L(\mathbf{W}\_{i-1})$ is the loss value, $\alpha$ the learning rate and $\nabla L(\mathbf{W}\_{i-1})$ the gradient of the loss.
  
  So, how do we get to the $L(\mathbf{W}\_{i-1})$ to calculate the gradient of $L(\mathbf{W}\_{i-1})$? As an example, we can initialize the set of $\mathbf{W}$ to 0.5. How can you explain it to me?
  
  **Tags:** deep-learning, gradient-descent
  
  > **Comments:**
  > **username_1**: Are you asking how to calculate the gradient of $f$ or how to calculate the loss value itself?  
  > **username_0**: Wow, Thanks for your edition. Before I can to calculate the gradient of f(W), I should have the f function. So, how can I have the f function at the first?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-12-29. Score: 2 | Accepted answer
  
  In your case, $L$ is the loss (or cost) function, which can be, for example, the [mean squared error (MSE)](https://en.wikipedia.org/wiki/Mean_squared_error) or the cross-entropy, depending on the problem you want to solve. Given one training example $(\mathbf{x}\_i, y\_i) \in D$, where $\mathbf{x}\_i \in \mathbb{R}^d$ is the input (for example, an image) and $y\_i \in \mathbb{R}$ can either be a label (aka class) or a numerical value, and $D$ is your training dataset, then the [MSE](https://en.wikipedia.org/wiki/Mean_squared_error) is defined as follows
  
  $$L(\mathbf{W}) = \frac{1}{2} \left(f(\mathbf{x}\_i) - y\_i \right)^2,$$
  
  where $f(\mathbf{x}\_i) \in \mathbb{R}$ is the output of the neural network $f$ given the input $\mathbf{x}\_i$.
  
  If you have a mini-batch of $M$ training examples $\{(\mathbf{x}\_i, y\_i) \}\_{i=1}^M$, then the loss will be an average of the MSE for each training example. For more info, have a look at this answer <https://ai.stackexchange.com/a/11675/2444>. The <https://ai.stackexchange.com/a/8985/2444> may also be useful.
  
  See the article [Loss and Loss Functions for Training Deep Learning Neural Networks](https://machinelearningmastery.com/loss-and-loss-functions-for-training-deep-learning-neural-networks/) for more info regarding different losses used in deep learning and how to choose the appropriate loss for your problem.
  
  > **Comments:**
  > **username_0**: Is that true to say : L(w) = 1/2 * (f(x,w) - y)**2 ?  
  > **username_0**: I did not get my answer completely however, your answer was very helpful. You said in this answer: ( https://ai.stackexchange.com/questions/11667/is-back-propagation-applied-for-each-data-point-or-for-a-batch-of-data-points/11675#11675 ) "For simplicity, assume that we are able to calculate the gradient of L, that is, delat(L)". So, how can I calculate the gradient of L function according to W set, because I need to update my W set. I mean that, do we have such a function like f(x) = 2x and then calculate the L(w)=1/2*(2x-y)**2 and delta(L) = 2*(2x-y)?  
  > **username_1**: $f(x, w)$ or $f(x)$ are "just" notations, in the sense that $w$ is clearly a parameter (or weight) of the neural network $x$, but $w$ is not an input to the neural network: $x$ is the input! The way you calculate the gradient of $L$ with respect to any of the parameters is with **back-propagation** and it depends on the architecture of the neural network and the loss function you choose. To understand back-propagation, you need to understand the basics of calculus.  
  > **username_0**: Thanks and I got the point : https://towardsdatascience.com/linear-regression-using-gradient-descent-97a6c8700931  
  
  ---
  
  