Answers.Body: <p>Well, you are definitely mixing two different things. Here are those bits:</p>
  
  <ul>
  <li><p>The function that deep learning approximates is basically a function that best fits the INPUT DATA points. You should not think about its differentiability or optimization aspects. We don't care what type of function it is; we just want the best fit of input data (ofcourse overfitting is not desired though). So, <strong>the space in which this fucntion lies has dimensions equal to the dimensions of input data (or number of input features)</strong>. For every function fit we get some loss which basically is the distance from actual data point from the one predicted by the function fitted.</p></li>
  <li><p>Loss function is the one which is defined in terms weights and biases. You can think <strong>the space in which loss function lies has dimensions equal to the number of PARAMETERS to tune (weights and biases)</strong>. So, this space is very different from the one described in the first point. Now, here we do care about the differentiability of the function because we want a point where the function outputs least value (minimize the loss) given some particular values of inputs (weights and biases in this case) and by differentiating we can traverse to that "optimal" point in the function.</p></li>
  </ul>
  
  <p>Hope this clears your doubt.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-27T20:24:47.857Z
Answers.Text: I think only really the second bullet is correct--if i generate a bunch of (x,y) data where y is some polynomial of x plus noise, then the function to be fit is really still in the set of polynomials--it doesn't really matter how many (x,y) points I make. Of course, regularization matters (how high of polynomial order should we permit?) and tanh or relu might not be the most efficient basis to approximate a polynomial, but, in the end, the only "dimension" that really matters is the number of knobs on the fitted function (weights and biases, or polynomial coefficients, or Fourier modes).
Answers.UserDisplayName: tsbertalan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-24T05:43:56.133Z
Answers.LastActivityDate: 2018-05-24T05:43:56.133Z
Answers.UserDisplayName: mausamsion
Body: <p>I was just reading through some convex optimization textbooks to hopefully improve my deep learning understanding and come up with new ideas. Halfway through, I decided to Google a bit! It's obvious that deep learning deals with nonconvex functions.</p>
  <p>Here's the question though: If deep learning is non-convex, then why do we apply a convex loss function, such as cross-entropy or least square, to solve a problem under a convex constraint? What am I missing?</p>
  
Tags: <machine-learning><deep-learning><optimization>
Title: If Deep Learning is non convex, then why do use a convex loss function?
UserDisplayName: Amir Hossein F
ThreadText: # If Deep Learning is non convex, then why do use a convex loss function?
  
  (Asked by: username_0 on 2018-05-23)
  
  I was just reading through some convex optimization textbooks to hopefully improve my deep learning understanding and come up with new ideas. Halfway through, I decided to Google a bit! It's obvious that deep learning deals with nonconvex functions.
  
  Here's the question though: If deep learning is non-convex, then why do we apply a convex loss function, such as cross-entropy or least square, to solve a problem under a convex constraint? What am I missing?
  
  **Tags:** machine-learning, deep-learning, optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-05-24. Score: 6 | Accepted answer
  
  Well, you are definitely mixing two different things. Here are those bits:
  
  * The function that deep learning approximates is basically a function that best fits the INPUT DATA points. You should not think about its differentiability or optimization aspects. We don't care what type of function it is; we just want the best fit of input data (ofcourse overfitting is not desired though). So, **the space in which this fucntion lies has dimensions equal to the dimensions of input data (or number of input features)**. For every function fit we get some loss which basically is the distance from actual data point from the one predicted by the function fitted.
  * Loss function is the one which is defined in terms weights and biases. You can think **the space in which loss function lies has dimensions equal to the number of PARAMETERS to tune (weights and biases)**. So, this space is very different from the one described in the first point. Now, here we do care about the differentiability of the function because we want a point where the function outputs least value (minimize the loss) given some particular values of inputs (weights and biases in this case) and by differentiating we can traverse to that "optimal" point in the function.
  
  Hope this clears your doubt.
  
  > **Comments:**
  > **username_2**: I think only really the second bullet is correct--if i generate a bunch of (x,y) data where y is some polynomial of x plus noise, then the function to be fit is really still in the set of polynomials--it doesn't really matter how many (x,y) points I make. Of course, regularization matters (how high of polynomial order should we permit?) and tanh or relu might not be the most efficient basis to approximate a polynomial, but, in the end, the only "dimension" that really matters is the number of knobs on the fitted function (weights and biases, or polynomial coefficients, or Fourier modes).  
  
  ---
  
  