Answers.Body: <blockquote>
  <p>So how does the total loss that I calculated after a forward pass even matter? Is it simply for logging(keeping track of)?</p>
  </blockquote>
  <p>Decreasing total loss over gradient update steps indicates that the model is learning. For calculating the individual gradients, what matters though are the current values in the nodes. (Since <span class="math-container">$\frac{\partial L}{\partial L} = 1.$</span>)</p>
  <blockquote>
  <p>Another doubt that I have is: What exactly is this graph indicating? If it is the Loss value for different set of parameter values, then shouldn't it be a discrete function?</p>
  </blockquote>
  <p>No, loss functions are generally continuous with respect to the parameter values. Imagine this as tuning the knob on parameter values, this results a continuous change in the loss. Recall that neural networks are essentially function compositions, and the layers used in practice are all continuous.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-10T10:46:27.947Z
Answers.Text: I get it now. Let's say our loss function was MSE. Then:
  L = (Y - Y')^2. Now the differential wrt any parameter would be 2(Y-Y') where Y is a constant (since expected output) and Y' is the NN function built at the back of differential functions(inherently continuous) parameterized by W in "n" dimensions. So essentially the loss function is a continuous function in "n" dimensions?
Answers.UserDisplayName: xkcd101
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-21T19:30:59.230Z
Answers.LastActivityDate: 2024-03-21T19:30:59.230Z
Answers.UserDisplayName: Ron
Body: <p>I'd say I have some understanding of backpropagation, however I am not really sure of the total loss being calculated.
  <a href="https://theneuralblog.com/forward-pass-backpropagation-example/" rel="nofollow noreferrer">Let us take the example below</a>:
  <a href="https://i.sstatic.net/zw2yj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zw2yj.png" alt="enter image description here" /></a></p>
  <p>After 1 forward pass when I have to update the parameters, I basically have to find answers to the question - What is the change in loss with respect to the change in a certain parameter i.e. if I were to update parameter w5 then, I need to calculate
  <span class="math-container">$$
  \frac{\partial L}{\partial w_5} = \frac{\partial L}{\partial o_1} \cdot \frac{\partial o_1}{\partial w_5}
  $$</span>
  So how does the total loss that I calculated after a forward pass even matter? Is it simply for logging(keeping track of)?</p>
  <p>Another doubt that I have is:
  What exactly is this graph indicating? If it is the Loss value for different set of parameter values, then shouldn't it be a discrete function?
  <a href="https://i.sstatic.net/YC7WW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/YC7WW.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-21T02:26:43.167Z
Comments.Text: "So how does the total loss that I calculated after a forward pass even matter?" what do you mean by this? How would you go about computing the partial derivative values without the loss?
Comments.UserDisplayName: Karl
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-10-10T10:41:26.077Z
Comments.Text: What I meant by total loss was the explicit numeric value. Let's say in a regression task, my target tensor to an MLP was [1,1,1] and my model returns the values [0.4,0.5,0.6]. The total loss here would be 0.36 + 0.25 + 0.16 = 0.77.
  However, in a computational graph, incase of automatic differentiation I shall be storing the local gradients based on the function differentials that would depend on the parameters affecting the local node. Hence, in the chain rule, I am not really going to use the value of Loss = 0.77. This is what I meant to ask if we're really using the total loss value or not
Comments.UserDisplayName: xkcd101
Tags: <neural-networks><backpropagation><multilayer-perceptrons>
Title: Total loss in backpropagation
UserDisplayName: xkcd101
ThreadText: # Total loss in backpropagation
  
  (Asked by: username_0 on 2024-03-20)
  
  I'd say I have some understanding of backpropagation, however I am not really sure of the total loss being calculated.
  [Let us take the example below](https://theneuralblog.com/forward-pass-backpropagation-example/):
  [![enter image description here](https://i.sstatic.net/zw2yj.png)](https://i.sstatic.net/zw2yj.png)
  
  After 1 forward pass when I have to update the parameters, I basically have to find answers to the question - What is the change in loss with respect to the change in a certain parameter i.e. if I were to update parameter w5 then, I need to calculate
  $$
  \frac{\partial L}{\partial w\_5} = \frac{\partial L}{\partial o\_1} \cdot \frac{\partial o\_1}{\partial w\_5}
  $$
  So how does the total loss that I calculated after a forward pass even matter? Is it simply for logging(keeping track of)?
  
  Another doubt that I have is:
  What exactly is this graph indicating? If it is the Loss value for different set of parameter values, then shouldn't it be a discrete function?
  [![enter image description here](https://i.sstatic.net/YC7WW.png)](https://i.sstatic.net/YC7WW.png)
  
  **Tags:** neural-networks, backpropagation, multilayer-perceptrons
  > **Comments:**
  > **username_1**: "So how does the total loss that I calculated after a forward pass even matter?" what do you mean by this? How would you go about computing the partial derivative values without the loss?  
  > **username_0**: What I meant by total loss was the explicit numeric value. Let's say in a regression task, my target tensor to an MLP was [1,1,1] and my model returns the values [0.4,0.5,0.6]. The total loss here would be 0.36 + 0.25 + 0.16 = 0.77. However, in a computational graph, incase of automatic differentiation I shall be storing the local gradients based on the function differentials that would depend on the parameters affecting the local node. Hence, in the chain rule, I am not really going to use the value of Loss = 0.77. This is what I meant to ask if we're really using the total loss value or not  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-03-21. Score: 0
  
  > So how does the total loss that I calculated after a forward pass even matter? Is it simply for logging(keeping track of)?
  
  Decreasing total loss over gradient update steps indicates that the model is learning. For calculating the individual gradients, what matters though are the current values in the nodes. (Since $\frac{\partial L}{\partial L} = 1.$)
  
  > Another doubt that I have is: What exactly is this graph indicating? If it is the Loss value for different set of parameter values, then shouldn't it be a discrete function?
  
  No, loss functions are generally continuous with respect to the parameter values. Imagine this as tuning the knob on parameter values, this results a continuous change in the loss. Recall that neural networks are essentially function compositions, and the layers used in practice are all continuous.
  
  > **Comments:**
  > **username_0**: I get it now. Let's say our loss function was MSE. Then: L = (Y - Y')^2. Now the differential wrt any parameter would be 2(Y-Y') where Y is a constant (since expected output) and Y' is the NN function built at the back of differential functions(inherently continuous) parameterized by W in "n" dimensions. So essentially the loss function is a continuous function in "n" dimensions?  
  
  ---
  
  