Answers.Body: <p>Let's take in consideration linear regression. You have a dataset composed by <span class="math-container">$x,y$</span> pairs, and you assume they are linearly related, thus you model this problem with LR:
  <span class="math-container">$$
  y = wx+b
  $$</span>
  Now, you want to find the <span class="math-container">$w$</span> and <span class="math-container">$b$</span> that best describe your data, thus you set a loss function, say MSE, and you minimize it:
  <span class="math-container">$$
  L(w, b) = \sum_{(x,y)\in D} (y - (wx+b))^2
  $$</span></p>
  <p>As you can see if you consider a single sample, this is a parabola. However, the fact that is a parabola, it's given by the loss function, and your model</p>
  <p>At that point, you try to minimize the average loss (the <span class="math-container">$1/n$</span> is discarded because it does not effect the minimization), so say you take &quot;the average parabola across the dataset&quot;</p>
  <p>However, you can clearly see that there is no relation between points in your dataset, as they are assumed to be independent, thus their loss function can be as far as they want (they are still parabolas, but very far from each other)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-13T12:38:01.570Z
Answers.Text: Assuming  we have a good and a representative dataset, and assuming that there is a good correlation between predictor features and the predicted feature, can we say that the global minimums of each loss surfaces should overlap and fall in same coordinates of loss surfaces? ( I mean a theoretical case where i keep backpropagating and  surching for global minimum of each loss surface whithout moving to.another observation till done with the current)
Answers.UserDisplayName: Igor
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-13T22:38:36.860Z
Answers.Text: @Igor given a distance metric (loss), your global minima would be by definition $L=0$, which means that the requirement is that the function approximated by your model intersects all the data in your dataset... however, there are infinite functions that can pass from N points, no matter the dimensionality, and the parameters of such functions might be arbitrarily far, thus global optima (s) do not overlap each other
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-10T11:47:02.570Z
Answers.LastActivityDate: 2023-08-10T11:47:02.570Z
Answers.UserDisplayName: Alberto
Body: <p>If I understand right that each observation whithin a dataset, creates a different loss surface where we want to find the global minimum.</p>
  <p>How different those surfaces one from another?
  Would it be correct to say that they differ like (for example) those two parabolas: f(x) = 5x^2 +4x+2  versus
  f(x) = 5x^2 +1x+8 which can be seen as same parabola located in another place of xy plane.</p>
  <p>Thank you</p>
  
Tags: <datasets><gradient-descent><loss><observation-spaces>
Title: What are the differences between loss surfaces that "derive"from different observations?
UserDisplayName: Igor
ThreadText: # What are the differences between loss surfaces that "derive"from different observations?
  
  (Asked by: username_0 on 2023-08-10)
  
  If I understand right that each observation whithin a dataset, creates a different loss surface where we want to find the global minimum.
  
  How different those surfaces one from another?
  Would it be correct to say that they differ like (for example) those two parabolas: f(x) = 5x^2 +4x+2 versus
  f(x) = 5x^2 +1x+8 which can be seen as same parabola located in another place of xy plane.
  
  Thank you
  
  **Tags:** datasets, gradient-descent, loss, observation-spaces
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-08-10. Score: 2 | Accepted answer
  
  Let's take in consideration linear regression. You have a dataset composed by $x,y$ pairs, and you assume they are linearly related, thus you model this problem with LR:
  $$
  y = wx+b
  $$
  Now, you want to find the $w$ and $b$ that best describe your data, thus you set a loss function, say MSE, and you minimize it:
  $$
  L(w, b) = \sum\_{(x,y)\in D} (y - (wx+b))^2
  $$
  
  As you can see if you consider a single sample, this is a parabola. However, the fact that is a parabola, it's given by the loss function, and your model
  
  At that point, you try to minimize the average loss (the $1/n$ is discarded because it does not effect the minimization), so say you take "the average parabola across the dataset"
  
  However, you can clearly see that there is no relation between points in your dataset, as they are assumed to be independent, thus their loss function can be as far as they want (they are still parabolas, but very far from each other)
  
  > **Comments:**
  > **username_0**: Assuming  we have a good and a representative dataset, and assuming that there is a good correlation between predictor features and the predicted feature, can we say that the global minimums of each loss surfaces should overlap and fall in same coordinates of loss surfaces? ( I mean a theoretical case where i keep backpropagating and  surching for global minimum of each loss surface whithout moving to.another observation till done with the current)  
  > **username_1**: @Igor given a distance metric (loss), your global minima would be by definition $L=0$, which means that the requirement is that the function approximated by your model intersects all the data in your dataset... however, there are infinite functions that can pass from N points, no matter the dimensionality, and the parameters of such functions might be arbitrarily far, thus global optima (s) do not overlap each other  
  
  ---
  
  