Answers.Body: <p>It is the same as other functions. You can use Theorem 2 in <a href="https://www.princeton.edu/%7Eaaa/Public/Teaching/ORF523/S16/ORF523_S16_Lec7_gh.pdf" rel="nofollow noreferrer">this lecture</a> (from Princeton University):</p>
  <p><a href="https://i.sstatic.net/AL3yc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AL3yc.png" alt="enter image description here" /></a></p>
  <p>(ii) condition is about the first-order condition for convexity and (iii) is the second-order. You can also find more detail in chapter 3 of <a href="https://stanford.edu/%7Eboyd/cvxbook/bv_cvxbook.pdf" rel="nofollow noreferrer">this book</a> (&quot;Convex Optimization&quot; by Stephen Boyd and Lieven Vandenberghe).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-02T23:12:54.677Z
Answers.Text: Please check that the lecture page is not accessible.
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-02T23:52:20.220Z
Answers.Text: @hanugm sorry, I've already checked and I can't see the problem.
Answers.UserDisplayName: OmG
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-03T00:08:46.453Z
Answers.Text: oh, no problem. It is showing **Error 404 Page Not Found Sorry, the page you are looking for is not available** for me
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-03T13:21:40.353Z
Answers.Text: @hanugm Sorry, it's strange. By the way, all links work for me.
Answers.UserDisplayName: OmG
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-20T21:17:31.883Z
Answers.Text: @OmG It also shows a 404 for me.
Answers.UserDisplayName: Squanchy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-21T08:13:24.203Z
Answers.Text: @Squanchy it's loaded for me.
Answers.UserDisplayName: OmG
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-21T10:24:48.210Z
Answers.Text: @hanugm I'm not sure what's up with the link... it's strange. If I click it, I indeed also get the error page. But if I then, in my browser, click the address bar and just immediately hit enter again without changing anything, it works fine. Directly copying the URL that OmG provided in their post, and pasting it in my browser, also works fine. So.. I'm not sure how it could be fixed, but it mostly works with a small workaround.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-22T02:34:56.133Z
Answers.Text: There is an issue with the URL encoding it is using %7E instead of ~.  You can see the difference in the GET request. This link should work fine https://www.princeton.edu/~aaa/Public/Teaching/ORF523/S16/ORF523_S16_Lec7_gh.pdf
Answers.UserDisplayName: Squanchy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-02T18:40:19.713Z
Answers.LastActivityDate: 2021-08-02T18:40:19.713Z
Answers.UserDisplayName: OmG
Answers.Body: <p>For the case of at least twice differentiable functions, the answer is given by @OmG - you need to look at the eigenvalues of Hessian.</p>
  <p>For the 1-dimensional case the picture is rather intuitive:</p>
  <p><a href="https://i.sstatic.net/h8QBJ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/h8QBJ.png" alt="enter image description here" /></a></p>
  <p>If the function grows faster then linear with deviation from the minimum, then the function is convex.</p>
  <p>For multidimensional case for any projection on a plane, cutting the loss hypersurface you need to have the same picture as for 1-dimensional case. This is satisfied, provided, that the Hessian matrix is positive definite.</p>
  <p><a href="https://i.sstatic.net/Te5Nm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Te5Nm.png" alt="enter image description here" /></a></p>
  <p>If the function is not twice differentiable, the situation becomes more complicated, and I am not aware of any general algorithm.</p>
  <p>However, for the case of piecewise-linear function, you can imagine something like <span class="math-container">$|x|$</span>, or a more general function like:
  <span class="math-container">$$
  f(x) = 
  \begin{cases}
  \alpha x &amp; \alpha &gt; 0, x &gt; 0 \\
  \beta x &amp; \beta &lt; 0, x &lt; 0
  \end{cases}
  $$</span></p>
  <p>If any section of the loss hypersurface in the vicinity of such a singularity has this form - the function will be convex as well.</p>
  <p><a href="https://i.sstatic.net/MKOp6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/MKOp6.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-03T13:22:08.730Z
Answers.LastActivityDate: 2021-08-03T13:38:22.373Z
Answers.LastEditDate: 2021-08-03T13:38:22.373Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>Loss functions are useful in calculating loss and then we can update the weights of a neural network. The loss function is thus useful in training neural networks.</p>
  <p>Consider the following excerpt from <a href="https://ai.stackexchange.com/questions/28877/what-are-the-necessary-mathematical-properties-to-be-a-loss-function-in-gradient">this</a> answer</p>
  <blockquote>
  <p>In principle, differentiability is sufficient to run gradient descent. <strong>That said, unless <span class="math-container">$L$</span> is convex, gradient descent offers no guarantees of convergence to a global minimiser</strong>. In practice, neural network loss functions are rarely convex anyway.</p>
  </blockquote>
  <p>It implies that the convexity property of loss functions is useful in ensuring the convergence, if we are using the gradient descent algorithm. There is <a href="https://ai.stackexchange.com/questions/28288/in-logistic-regression-why-is-the-binary-cross-entropy-loss-function-convex">another narrowed version of this question</a> dealing with cross-entropy loss. But, this question is, in fact, a general question and is not restricted to a particular loss function.</p>
  <p>How to know whether a loss function is convex or not? Is there any algorithm to check it?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-03T13:04:49.960Z
Comments.Text: [Here](https://ai.stackexchange.com/q/11979/2444) is a related question.
Comments.UserDisplayName: nbro
Tags: <objective-functions><convergence><convex-function>
Title: How to check whether my loss function is convex or not?
UserDisplayName: hanugm
ThreadText: # How to check whether my loss function is convex or not?
  
  (Asked by: username_0 on 2021-08-02)
  
  Loss functions are useful in calculating loss and then we can update the weights of a neural network. The loss function is thus useful in training neural networks.
  
  Consider the following excerpt from [this](https://ai.stackexchange.com/questions/28877/what-are-the-necessary-mathematical-properties-to-be-a-loss-function-in-gradient) answer
  
  > In principle, differentiability is sufficient to run gradient descent. **That said, unless $L$ is convex, gradient descent offers no guarantees of convergence to a global minimiser**. In practice, neural network loss functions are rarely convex anyway.
  
  It implies that the convexity property of loss functions is useful in ensuring the convergence, if we are using the gradient descent algorithm. There is [another narrowed version of this question](https://ai.stackexchange.com/questions/28288/in-logistic-regression-why-is-the-binary-cross-entropy-loss-function-convex) dealing with cross-entropy loss. But, this question is, in fact, a general question and is not restricted to a particular loss function.
  
  How to know whether a loss function is convex or not? Is there any algorithm to check it?
  
  **Tags:** objective-functions, convergence, convex-function
  > **Comments:**
  > **username_1**: [Here](https://ai.stackexchange.com/q/11979/2444) is a related question.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-08-02. Score: 2
  
  It is the same as other functions. You can use Theorem 2 in [this lecture](https://www.princeton.edu/%7Eaaa/Public/Teaching/ORF523/S16/ORF523_S16_Lec7_gh.pdf) (from Princeton University):
  
  [![enter image description here](https://i.sstatic.net/AL3yc.png)](https://i.sstatic.net/AL3yc.png)
  
  (ii) condition is about the first-order condition for convexity and (iii) is the second-order. You can also find more detail in chapter 3 of [this book](https://stanford.edu/%7Eboyd/cvxbook/bv_cvxbook.pdf) ("Convex Optimization" by Stephen Boyd and Lieven Vandenberghe).
  
  > **Comments:**
  > **username_0**: Please check that the lecture page is not accessible.  
  > **username_2**: @hanugm sorry, I've already checked and I can't see the problem.  
  > **username_0**: oh, no problem. It is showing **Error 404 Page Not Found Sorry, the page you are looking for is not available** for me  
  > **username_2**: @hanugm Sorry, it's strange. By the way, all links work for me.  
  > **username_3**: @OmG It also shows a 404 for me.  
  > **username_2**: @Squanchy it's loaded for me.  
  > **username_4**: @hanugm I'm not sure what's up with the link... it's strange. If I click it, I indeed also get the error page. But if I then, in my browser, click the address bar and just immediately hit enter again without changing anything, it works fine. Directly copying the URL that OmG provided in their post, and pasting it in my browser, also works fine. So.. I'm not sure how it could be fixed, but it mostly works with a small workaround.  
  > **username_3**: There is an issue with the URL encoding it is using %7E instead of ~.  You can see the difference in the GET request. This link should work fine https://www.princeton.edu/~aaa/Public/Teaching/ORF523/S16/ORF523_S16_Lec7_gh.pdf  
  
  ---
  
  ### Answer by username_5 on 2021-08-03. Score: 2
  
  For the case of at least twice differentiable functions, the answer is given by @OmG - you need to look at the eigenvalues of Hessian.
  
  For the 1-dimensional case the picture is rather intuitive:
  
  [![enter image description here](https://i.sstatic.net/h8QBJ.png)](https://i.sstatic.net/h8QBJ.png)
  
  If the function grows faster then linear with deviation from the minimum, then the function is convex.
  
  For multidimensional case for any projection on a plane, cutting the loss hypersurface you need to have the same picture as for 1-dimensional case. This is satisfied, provided, that the Hessian matrix is positive definite.
  
  [![enter image description here](https://i.sstatic.net/Te5Nm.png)](https://i.sstatic.net/Te5Nm.png)
  
  If the function is not twice differentiable, the situation becomes more complicated, and I am not aware of any general algorithm.
  
  However, for the case of piecewise-linear function, you can imagine something like $|x|$, or a more general function like:
  $$
  f(x) =
  \begin{cases}
  \alpha x & \alpha > 0, x > 0 \\
  \beta x & \beta < 0, x < 0
  \end{cases}
  $$
  
  If any section of the loss hypersurface in the vicinity of such a singularity has this form - the function will be convex as well.
  
  [![enter image description here](https://i.sstatic.net/MKOp6.png)](https://i.sstatic.net/MKOp6.png)
  
  
  ---
  
  