Answers.Body: <p>Answer in short:  MSE is convex on its input and parameters by itself. But on an arbitrary neural network it is not always convex due to the presence of non-linearities in the form of activation functions. Source for my answer is <a href="https://math.stackexchange.com/questions/2402455/convexity-of-mse-in-neural-networks/2402472">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-14T06:09:55.703Z
Answers.LastActivityDate: 2018-08-14T06:09:55.703Z
Answers.UserDisplayName: varsh
Answers.Body: <p><strong>Convexity</strong></p>
  
  <blockquote>
    <p>A function <span class="math-container">$f(x)$</span> with <span class="math-container">$x ∈ Χ$</span> is convex, if, for any <span class="math-container">$x_1 ∈ Χ$</span>, <span class="math-container">$x_2 ∈ Χ$</span> and for any <span class="math-container">$0 ≤ λ ≤ 1$</span>,
    <span class="math-container">$$f(λ x_1 + (1 − λ) x_2) ≤ λf(x_1) + (1 − λ) f (x_2).$$</span></p>
  </blockquote>
  
  <p>It can be proven that such convex <span class="math-container">$f(x)$</span> has one global minimum.  A unique global minimum eliminates traps created by local minima that can occur in algorithms that attempt to achieve convergence on a global minimum, such as the minimization of an error function.</p>
  
  <p>Although an error function may be 100% reliable in all continuous, linear contexts and many non-linear contexts, it does not mean the convergence on a global minimum for all possible non-linear contexts.</p>
  
  <p><strong>Mean Square Error</strong></p>
  
  <p>Given a function <span class="math-container">$s(x)$</span> describing ideal system behavior and a model of the system <span class="math-container">$a(x, p)$</span> (where <span class="math-container">$p$</span> is the parameter vector, matrix, cube, or hypercube and <span class="math-container">$1 ≤ n ≤ N$</span>), created rationally or via convergence (as in neural net training), the mean square error (MSE) function can be represented as follows.</p>
  
  <p><span class="math-container">$$e(β) := N^{-1} \sum_{n} [a(x_n) − s(x_n)]^2$$</span></p>
  
  <p>The material you are reading is probably not claiming that <span class="math-container">$a(x, p)$</span> or <span class="math-container">$s(x)$</span> are convex with respect to <span class="math-container">$x$</span>, but that <span class="math-container">$e(β)$</span> is convex with respect to <span class="math-container">$a(x, p)$</span> and <span class="math-container">$s(x)$</span> no matter what they are.  This later statement can be proven for any continuous <span class="math-container">$a(x, p)$</span> and <span class="math-container">$s(x)$</span>.  </p>
  
  <p><strong>Confounding the Convergence Algorithm</strong></p>
  
  <p>If the question is whether a specific <span class="math-container">$a(x, p)$</span> and method of achieving an <span class="math-container">$s(x)$</span> that approximates the <span class="math-container">$a(x, p)$</span> within a reasonable MSE convergence margin can be confounded, the answer is, "Yes."  That is why MSE is not the only error model.</p>
  
  <p><strong>Summary</strong></p>
  
  <p>The best way summarize is that <span class="math-container">$e(β)$</span> should be defined or chosen from a set of stock convex error models based on the following knowledge.</p>
  
  <ul>
  <li>Known properties of the system <span class="math-container">$s(x)$</span></li>
  <li>The definition of the approximation model <span class="math-container">$a(x, p)$</span></li>
  <li>Tensor used to generate the next state in the convergent sequence</li>
  </ul>
  
  <p>The set of stock convex error models certainly includes the MSE model because of its simplicity and computational thrift.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-08-23T10:31:35.500Z
Answers.LastActivityDate: 2018-11-22T06:58:43.990Z
Answers.LastEditDate: 2018-11-22T06:58:43.990Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>Multiple resources I referred to mention that MSE is great because it's convex. But I don't get how, especially in the context of neural networks.</p>
  
  <p>Let's say we have the following:</p>
  
  <ul>
  <li><span class="math-container">$X$</span>: training dataset</li>
  <li><span class="math-container">$Y$</span>: targets</li>
  <li><span class="math-container">$\Theta$</span>: the set of parameters of the model <span class="math-container">$f_\Theta$</span> (a neural network model with non-linearities)</li>
  </ul>
  
  <p>Then:</p>
  
  <p><span class="math-container">$$\operatorname{MSE}(\Theta) = (f_\Theta(X) - Y)^2$$</span></p>
  
  <p>Why would this loss function always be convex? Does this depend on <span class="math-container">$f_\Theta(X)$</span>?</p>
  
Tags: <neural-networks><math><backpropagation><gradient-descent>
Title: Is the mean-squared error always convex in the context of neural networks?
UserDisplayName: user74211
ThreadText: # Is the mean-squared error always convex in the context of neural networks?
  
  (Asked by: username_0 on 2017-08-22)
  
  Multiple resources I referred to mention that MSE is great because it's convex. But I don't get how, especially in the context of neural networks.
  
  Let's say we have the following:
  
  * $X$: training dataset
  * $Y$: targets
  * $\Theta$: the set of parameters of the model $f\_\Theta$ (a neural network model with non-linearities)
  
  Then:
  
  $$\operatorname{MSE}(\Theta) = (f\_\Theta(X) - Y)^2$$
  
  Why would this loss function always be convex? Does this depend on $f\_\Theta(X)$?
  
  **Tags:** neural-networks, math, backpropagation, gradient-descent
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-08-14. Score: 5
  
  Answer in short: MSE is convex on its input and parameters by itself. But on an arbitrary neural network it is not always convex due to the presence of non-linearities in the form of activation functions. Source for my answer is [here](https://math.stackexchange.com/questions/2402455/convexity-of-mse-in-neural-networks/2402472).
  
  
  ---
  
  ### Answer by username_2 on 2017-08-23. Score: 3
  
  **Convexity**
  
  > A function $f(x)$ with $x ∈ Χ$ is convex, if, for any $x\_1 ∈ Χ$, $x\_2 ∈ Χ$ and for any $0 ≤ λ ≤ 1$,
  > $$f(λ x\_1 + (1 − λ) x\_2) ≤ λf(x\_1) + (1 − λ) f (x\_2).$$
  
  It can be proven that such convex $f(x)$ has one global minimum. A unique global minimum eliminates traps created by local minima that can occur in algorithms that attempt to achieve convergence on a global minimum, such as the minimization of an error function.
  
  Although an error function may be 100% reliable in all continuous, linear contexts and many non-linear contexts, it does not mean the convergence on a global minimum for all possible non-linear contexts.
  
  **Mean Square Error**
  
  Given a function $s(x)$ describing ideal system behavior and a model of the system $a(x, p)$ (where $p$ is the parameter vector, matrix, cube, or hypercube and $1 ≤ n ≤ N$), created rationally or via convergence (as in neural net training), the mean square error (MSE) function can be represented as follows.
  
  $$e(β) := N^{-1} \sum\_{n} [a(x\_n) − s(x\_n)]^2$$
  
  The material you are reading is probably not claiming that $a(x, p)$ or $s(x)$ are convex with respect to $x$, but that $e(β)$ is convex with respect to $a(x, p)$ and $s(x)$ no matter what they are. This later statement can be proven for any continuous $a(x, p)$ and $s(x)$.
  
  **Confounding the Convergence Algorithm**
  
  If the question is whether a specific $a(x, p)$ and method of achieving an $s(x)$ that approximates the $a(x, p)$ within a reasonable MSE convergence margin can be confounded, the answer is, "Yes." That is why MSE is not the only error model.
  
  **Summary**
  
  The best way summarize is that $e(β)$ should be defined or chosen from a set of stock convex error models based on the following knowledge.
  
  * Known properties of the system $s(x)$
  * The definition of the approximation model $a(x, p)$
  * Tensor used to generate the next state in the convergent sequence
  
  The set of stock convex error models certainly includes the MSE model because of its simplicity and computational thrift.
  
  
  ---
  
  