Answers.Body: <p>The <span class="math-container">$L_{CE}$</span> that you provided is binary cross-entropy, the factor <span class="math-container">$y$</span> and <span class="math-container">$(1-y)$</span> is because <span class="math-container">$y$</span> is binary <span class="math-container">$({0,1})$</span>, careful with the name next time. The cross-entropy loss should have form:</p>
  <p><span class="math-container">$$L_{CE}=-\displaystyle\sum_{i=1}^C y_i\log(\hat{y_i})$$</span></p>
  <p>Where <span class="math-container">$C$</span> is the number of classes. Normally, the <span class="math-container">$y_i$</span> factor only is 1 when <span class="math-container">$i$</span> is the index of the correct class. Therefore, with each class, the function is just:</p>
  <p><span class="math-container">$$f(x)=-log(x)$$</span></p>
  <p>Now, to prove this one is convex, we have multiple ways, but my favorite one is computing the derivative and second derivative.</p>
  <p><span class="math-container">$$\frac{\partial L}{\partial x}=-\frac{1}{x}\Rightarrow \frac{\partial^2 L}{\partial x^2}=\frac{1}{x^2}&gt;0 \text{ for all }x\in(0,1]$$</span></p>
  <p>This proves that <span class="math-container">$f(x) = -\log(x)$</span> is convex, given that, if the second derivative of a function is positive, then the function is convex (more info and an example <a href="https://www.hec.ca/en/cams/help/topics/The_second_derivative.pdf" rel="nofollow noreferrer">here</a>).</p>
  <p>For the case of multiple samples, we also need to prove that the sum of a convex function is a convex function.</p>
  <p>Based on <a href="https://en.wikipedia.org/wiki/Convex_function" rel="nofollow noreferrer">the definition of convex function</a>, a function <span class="math-container">$f:X\rightarrow \mathbb{R}$</span> will be convex if:
  <span class="math-container">$$f(tx_1+(1-t)x_2) \le tf(x_1) + (1-t)f(x_2)$$</span>
  where <span class="math-container">$0&lt;t&lt;1$</span> and <span class="math-container">$x_1,x_2\in X$</span>.</p>
  <p>Now, let's assume <span class="math-container">$h(x) = f(x) + g(x)$</span> where both <span class="math-container">$f$</span> and <span class="math-container">$g$</span> are convex. We have:
  <span class="math-container">$$f(tx_1+(1-t)x_2) \le tf(x_1) + (1-t)f(x_2)$$</span>
  <span class="math-container">$$g(tx_1+(1-t)x_2) \le tg(x_1) + (1-t)g(x_2)$$</span>
  <span class="math-container">$$\Rightarrow f(tx_1+(1-t)x_2) + g(tx_1+(1-t)x_2) \le tf(x_1) + (1-t)f(x_2) + tg(x_1) + (1-t)g(x_2)$$</span>
  <span class="math-container">$$\Rightarrow f(tx_1+(1-t)x_2) + g(tx_1+(1-t)x_2) \le t(f(x_1)+g(x_1)) + (1-t)(f(x_2)+g(x_2))$$</span>
  <span class="math-container">$$\Rightarrow h(tx_1+(1-t)x_2) \le th(x_1) + (1-t)h(x_2)$$</span>
  <span class="math-container">$\Rightarrow h$</span> is the convex function <span class="math-container">$\Rightarrow$</span> the summation of convex functions is a convex function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-22T04:44:04.530Z
Answers.LastActivityDate: 2021-12-23T11:36:07.127Z
Answers.LastEditDate: 2021-12-23T11:36:07.127Z
Answers.UserDisplayName: CuCaRot
Answers.Body: <p>If you find the Hessian matrix (the matrix of second order derivatives) for the binary cross entropy loss function, you'll see that it is positive semidefinite for any possible value of the parameters. This concludes that it is a convex function.</p>
  <p>A side effect of it being convex is that it will have a single minimum as mentioned in the textbook you cited.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-21T22:19:16.183Z
Answers.Text: Maybe you could at least provide a link to a reference that shows this more in detail. For people also not very familiar with linear algebra, you could also explain a little bit more the relationship between the convexivity and being or not positive semi-definite (and what this latter even means).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-21T11:29:31.443Z
Answers.LastActivityDate: 2021-06-21T11:29:31.443Z
Answers.UserDisplayName: Abhijith S. Raj
Answers.Body: <p>I'm unable to comment on previous answers because I'm new to ai.stackexchange and don't have enough clout points. So I'm writing my comment as an answer instead.</p>
  <p>Unless I'm missing something, I believe there are a couple errors in the answer given by @CuCaRot. First, the alternative formula he gave for <span class="math-container">$L_{CE}$</span> is partially incorrect because it should be <span class="math-container">$\log(1-\hat{y})$</span> in the case of class 0, and not <span class="math-container">$\log(\hat{y})$</span>. Second, the original question was about the convexity of the log loss as a function of the weights w, and not as a function of <span class="math-container">$\hat{y}$</span>; consequently the derivation used to prove convexity is flawed.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-19T16:07:40.753Z
Answers.Text: You can search for, and view, this text: "To make requests for clarification, or mere responses to other answers, you **need to wait** until you have the comment privilege." in the FAQ: "[Why do I need 50 reputation to comment? What can I do instead?](https://meta.stackexchange.com/a/214174/282094)".  --- See also: [Converting answer to comment](https://meta.stackexchange.com/a/74161/282094).
Answers.UserDisplayName: Rob
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-19T19:29:11.527Z
Answers.Text: I understand the rationale behind needing the minimum 50 points to be able to comment, but I'm hoping someone will see and agree with my "comment" and convert it into a real comment, since I can't do it.
Answers.UserDisplayName: Chiraz BenAbdelkader
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-19T20:24:57.377Z
Answers.Text: you need to flag your comment and ask (custom reason); it seems important enough that the information should be provided, but it can't be an "answer" and you should expect it to be deleted.
Answers.UserDisplayName: Rob
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-19T13:31:46.687Z
Answers.LastActivityDate: 2022-05-19T13:34:46.177Z
Answers.LastEditDate: 2022-05-19T13:34:46.177Z
Answers.UserDisplayName: Chiraz BenAbdelkader
Body: <p>I am studying logistic regression for binary classification.</p>
  <p>The loss function used is <strong>cross-entropy</strong>. For a given input <span class="math-container">$x$</span>, if our model outputs <span class="math-container">$\hat{y}$</span> instead of <span class="math-container">$y$</span>, the loss is given by
  <span class="math-container">$$\text{L}_{\text{CE}}(y,\hat{y}) =  -[y \log \hat{y} + (1 - y) (\log{1 - \hat{y}})]$$</span></p>
  <p>Suppose there are <span class="math-container">$m$</span> such training examples, then the overall total loss function <span class="math-container">$\text{TL}_{\text{CE}}$</span> is given by</p>
  <p><span class="math-container">$$\text{TL}_{\text{CE}} = \dfrac{1}{m} \sum\limits_{i = 1}^{m} \text{L}_{\text{CE}} (y_i , \hat{y_i}) $$</span></p>
  <p>It is said that <strong>the loss function is convex</strong>. That is, If I draw a graph between the loss values wrt the corresponding weights then the curve will be convex. The <a href="https://web.stanford.edu/%7Ejurafsky/slp3/5.pdf" rel="nofollow noreferrer">material from textbook</a> did not give any explanation regarding the convex nature of the cross-entropy loss function. You can observe it from the following passage.</p>
  <blockquote>
  <p>For logistic regression, <strong>this (cross-entropy) loss function is conveniently convex</strong>. A
  convex function has just one minimum; there are no local minima
  to get stuck in, so gradient descent starting from any point is
  guaranteed to find the minimum.  (By contrast, the loss for multi-layer
  neural networks is non-convex,  and gradient descent may get stuck in
  local minima for neural network training and never find the global
  optimum.)</p>
  </blockquote>
  <p>How did they conclude conveniently that the loss function is convex? Is it by plotting or some other means?</p>
  
Tags: <objective-functions><optimization><gradient-descent><binary-classification><cross-entropy>
Title: In logistic regression, why is the binary cross-entropy loss function convex?
UserDisplayName: hanugm
ThreadText: # In logistic regression, why is the binary cross-entropy loss function convex?
  
  (Asked by: username_0 on 2021-06-16)
  
  I am studying logistic regression for binary classification.
  
  The loss function used is **cross-entropy**. For a given input $x$, if our model outputs $\hat{y}$ instead of $y$, the loss is given by
  $$\text{L}\_{\text{CE}}(y,\hat{y}) = -[y \log \hat{y} + (1 - y) (\log{1 - \hat{y}})]$$
  
  Suppose there are $m$ such training examples, then the overall total loss function $\text{TL}\_{\text{CE}}$ is given by
  
  $$\text{TL}\_{\text{CE}} = \dfrac{1}{m} \sum\limits\_{i = 1}^{m} \text{L}\_{\text{CE}} (y\_i , \hat{y\_i}) $$
  
  It is said that **the loss function is convex**. That is, If I draw a graph between the loss values wrt the corresponding weights then the curve will be convex. The [material from textbook](https://web.stanford.edu/%7Ejurafsky/slp3/5.pdf) did not give any explanation regarding the convex nature of the cross-entropy loss function. You can observe it from the following passage.
  
  > For logistic regression, **this (cross-entropy) loss function is conveniently convex**. A
  > convex function has just one minimum; there are no local minima
  > to get stuck in, so gradient descent starting from any point is
  > guaranteed to find the minimum. (By contrast, the loss for multi-layer
  > neural networks is non-convex, and gradient descent may get stuck in
  > local minima for neural network training and never find the global
  > optimum.)
  
  How did they conclude conveniently that the loss function is convex? Is it by plotting or some other means?
  
  **Tags:** objective-functions, optimization, gradient-descent, binary-classification, cross-entropy
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2021-12-22. Score: 3
  
  The $L\_{CE}$ that you provided is binary cross-entropy, the factor $y$ and $(1-y)$ is because $y$ is binary $({0,1})$, careful with the name next time. The cross-entropy loss should have form:
  
  $$L\_{CE}=-\displaystyle\sum\_{i=1}^C y\_i\log(\hat{y\_i})$$
  
  Where $C$ is the number of classes. Normally, the $y\_i$ factor only is 1 when $i$ is the index of the correct class. Therefore, with each class, the function is just:
  
  $$f(x)=-log(x)$$
  
  Now, to prove this one is convex, we have multiple ways, but my favorite one is computing the derivative and second derivative.
  
  $$\frac{\partial L}{\partial x}=-\frac{1}{x}\Rightarrow \frac{\partial^2 L}{\partial x^2}=\frac{1}{x^2}>0 \text{ for all }x\in(0,1]$$
  
  This proves that $f(x) = -\log(x)$ is convex, given that, if the second derivative of a function is positive, then the function is convex (more info and an example [here](https://www.hec.ca/en/cams/help/topics/The_second_derivative.pdf)).
  
  For the case of multiple samples, we also need to prove that the sum of a convex function is a convex function.
  
  Based on [the definition of convex function](https://en.wikipedia.org/wiki/Convex_function), a function $f:X\rightarrow \mathbb{R}$ will be convex if:
  $$f(tx\_1+(1-t)x\_2) \le tf(x\_1) + (1-t)f(x\_2)$$
  where $0<t<1$ and $x\_1,x\_2\in X$.
  
  Now, let's assume $h(x) = f(x) + g(x)$ where both $f$ and $g$ are convex. We have:
  $$f(tx\_1+(1-t)x\_2) \le tf(x\_1) + (1-t)f(x\_2)$$
  $$g(tx\_1+(1-t)x\_2) \le tg(x\_1) + (1-t)g(x\_2)$$
  $$\Rightarrow f(tx\_1+(1-t)x\_2) + g(tx\_1+(1-t)x\_2) \le tf(x\_1) + (1-t)f(x\_2) + tg(x\_1) + (1-t)g(x\_2)$$
  $$\Rightarrow f(tx\_1+(1-t)x\_2) + g(tx\_1+(1-t)x\_2) \le t(f(x\_1)+g(x\_1)) + (1-t)(f(x\_2)+g(x\_2))$$
  $$\Rightarrow h(tx\_1+(1-t)x\_2) \le th(x\_1) + (1-t)h(x\_2)$$
  $\Rightarrow h$ is the convex function $\Rightarrow$ the summation of convex functions is a convex function.
  
  
  ---
  
  ### Answer by username_2 on 2021-06-21. Score: 1
  
  If you find the Hessian matrix (the matrix of second order derivatives) for the binary cross entropy loss function, you'll see that it is positive semidefinite for any possible value of the parameters. This concludes that it is a convex function.
  
  A side effect of it being convex is that it will have a single minimum as mentioned in the textbook you cited.
  
  > **Comments:**
  > **username_3**: Maybe you could at least provide a link to a reference that shows this more in detail. For people also not very familiar with linear algebra, you could also explain a little bit more the relationship between the convexivity and being or not positive semi-definite (and what this latter even means).  
  
  ---
  
  ### Answer by username_4 on 2022-05-19. Score: 1
  
  I'm unable to comment on previous answers because I'm new to ai.stackexchange and don't have enough clout points. So I'm writing my comment as an answer instead.
  
  Unless I'm missing something, I believe there are a couple errors in the answer given by @username_1. First, the alternative formula he gave for $L\_{CE}$ is partially incorrect because it should be $\log(1-\hat{y})$ in the case of class 0, and not $\log(\hat{y})$. Second, the original question was about the convexity of the log loss as a function of the weights w, and not as a function of $\hat{y}$; consequently the derivation used to prove convexity is flawed.
  
  > **Comments:**
  > **username_5**: You can search for, and view, this text: "To make requests for clarification, or mere responses to other answers, you **need to wait** until you have the comment privilege." in the FAQ: "[Why do I need 50 reputation to comment? What can I do instead?](https://meta.stackexchange.com/a/214174/282094)".  --- See also: [Converting answer to comment](https://meta.stackexchange.com/a/74161/282094).  
  > **username_4**: I understand the rationale behind needing the minimum 50 points to be able to comment, but I'm hoping someone will see and agree with my "comment" and convert it into a real comment, since I can't do it.  
  > **username_5**: you need to flag your comment and ask (custom reason); it seems important enough that the information should be provided, but it can't be an "answer" and you should expect it to be deleted.  
  
  ---
  
  