Answers.Body: <p>If I understood your question correctly - the quadratic loss function is not always convex. Its' convexity depends on the input it takes. For example, consider a very basic NN that takes some input <span class="math-container">$x$</span> and returns
  <span class="math-container">\begin{equation}
  f(x)=relu(w_1relu(w_0x+b_0)+b_1).
  \end{equation}</span>
  Using quadratic loss, or <span class="math-container">$\ell_2$</span> is minimizing the objective <span class="math-container">$\sum_i(x_i-f(x_i))^2$</span>, but even when examining only one of the terms (say <span class="math-container">$i=0$</span>), which is <span class="math-container">$(x_0-f(x_0))^2$</span>, we see that it is not convex at all, as the following plot shows:</p>
  <p>                                          <a href="https://i.sstatic.net/Dwte2.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Dwte2.png" alt="plot" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-16T15:01:17.780Z
Answers.Text: Yes, you did. So the convexity of the loss depends on both inputs (prediction_function and groundtruth_function) ? As soon as one is non-convex, the loss will also be non-convex right? What confuses me, I guess most of the real-world problems will be non-convex and thus would always lead to a non-convex loss, and I thought simple linear classifiers (1/1+e(-(wx+b))) are convex ? Or does the convexity just depends on (in your example) f(x_i) and not as well x_i ?
Answers.UserDisplayName: horsti
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-16T15:11:08.753Z
Answers.Text: Generally speaking, you are correct as the composition of a convex function with a non-convex one may very well be non-convex. Notice that a sigmoid function is not convex (think of it as an "S" shape)
Answers.UserDisplayName: Hadar Sharvit
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-16T15:31:22.997Z
Answers.Text: So the convexity of the loss is also depending on my underlying groundtruth function, which I try to approximate, right ? So it is very likely that a very complex task, which I try to solve with a linear classifier would yield to a non-convex loss-function (although I am using the BinaryCrossEntropy loss) ? Or does the loss-convexity just depend on the "prediction_function" ?
Answers.UserDisplayName: horsti
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-09T13:39:51.337Z
Answers.LastActivityDate: 2022-09-09T13:44:59.697Z
Answers.LastEditDate: 2022-09-09T13:44:59.697Z
Answers.UserDisplayName: Hadar Sharvit
Body: <p>there is one problem which bugs me quite a long time, it is the non-convex loss shape (multiple minima, e.g. shown <a href="https://medium.com/swlh/non-convex-optimization-in-deep-learning-26fa30a2b2b3" rel="nofollow noreferrer">here</a>) of neural networks which use a quadratic loss function.</p>
  <p>Question: Why is a “common” AI problem usually non-convex with multiple minima, although we are using e.g. quadratic loss functions (which is in lectures usually drawn as a simple,convex,quadratic function such as x^2)?</p>
  <p>My guess:</p>
  <p>Is it because we are feeding the loss function with our highly non-linear model-output and therefore the resulting total loss surface is highly non-linear/ non-convex? Specifically, the quadratic loss is just approximating the infinitesimal small neighbourhood around a specific point (minibatch) as quadratic. Is that guess correct? This would imply, that highly-non linear / very deep and complex models have a highly-non linear resulting loss-surface, while shallower models have less minima and a one-layer network has a convex shape ?</p>
  
Tags: <machine-learning><deep-learning><loss><convex-function>
Title: Non-Convex loss-surface although quadratic loss function
UserDisplayName: horsti
ThreadText: # Non-Convex loss-surface although quadratic loss function
  
  (Asked by: username_0 on 2022-09-09)
  
  there is one problem which bugs me quite a long time, it is the non-convex loss shape (multiple minima, e.g. shown [here](https://medium.com/swlh/non-convex-optimization-in-deep-learning-26fa30a2b2b3)) of neural networks which use a quadratic loss function.
  
  Question: Why is a “common” AI problem usually non-convex with multiple minima, although we are using e.g. quadratic loss functions (which is in lectures usually drawn as a simple,convex,quadratic function such as x^2)?
  
  My guess:
  
  Is it because we are feeding the loss function with our highly non-linear model-output and therefore the resulting total loss surface is highly non-linear/ non-convex? Specifically, the quadratic loss is just approximating the infinitesimal small neighbourhood around a specific point (minibatch) as quadratic. Is that guess correct? This would imply, that highly-non linear / very deep and complex models have a highly-non linear resulting loss-surface, while shallower models have less minima and a one-layer network has a convex shape ?
  
  **Tags:** machine-learning, deep-learning, loss, convex-function
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-09-09. Score: 1 | Accepted answer
  
  If I understood your question correctly - the quadratic loss function is not always convex. Its' convexity depends on the input it takes. For example, consider a very basic NN that takes some input $x$ and returns
  \begin{equation}
  f(x)=relu(w\_1relu(w\_0x+b\_0)+b\_1).
  \end{equation}
  Using quadratic loss, or $\ell\_2$ is minimizing the objective $\sum\_i(x\_i-f(x\_i))^2$, but even when examining only one of the terms (say $i=0$), which is $(x\_0-f(x\_0))^2$, we see that it is not convex at all, as the following plot shows:
  
  [![plot](https://i.sstatic.net/Dwte2.png)](https://i.sstatic.net/Dwte2.png)
  
  > **Comments:**
  > **username_0**: Yes, you did. So the convexity of the loss depends on both inputs (prediction_function and groundtruth_function) ? As soon as one is non-convex, the loss will also be non-convex right? What confuses me, I guess most of the real-world problems will be non-convex and thus would always lead to a non-convex loss, and I thought simple linear classifiers (1/1+e(-(wx+b))) are convex ? Or does the convexity just depends on (in your example) f(x_i) and not as well x_i ?  
  > **username_1**: Generally speaking, you are correct as the composition of a convex function with a non-convex one may very well be non-convex. Notice that a sigmoid function is not convex (think of it as an "S" shape)  
  > **username_0**: So the convexity of the loss is also depending on my underlying groundtruth function, which I try to approximate, right ? So it is very likely that a very complex task, which I try to solve with a linear classifier would yield to a non-convex loss-function (although I am using the BinaryCrossEntropy loss) ? Or does the loss-convexity just depend on the "prediction_function" ?  
  
  ---
  
  