Answers.Body: <p>You're missing a couple of quite important concepts:</p>
  <ul>
  <li><a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem" rel="nofollow noreferrer">Universal approximation theorem</a>: with enough parameters a neural network can approximate any function.</li>
  <li>Basically every loss function is non convex. (There is this little problem in machine learning call local minima about which we like to complain a lot :) )</li>
  </ul>
  <p>But no need to trust me, just run a simple experiment and try yourself to approximate a non convex function, like <strong><span class="math-container">$sin(x)$</span></strong> with relu:</p>
  <pre><code>from sklearn.neural_network import MLPRegressor
  import numpy as np
  import matplotlib.pyplot as plt
  
  f = lambda x: [[x_] for x_ in x]
  noise_level = 0.1
  X_train_ = np.arange(0, 10, 0.2)
  real_sin = np.sin(X_train_)
  y_train = real_sin + np.random.normal(0, noise_level, len(X_train_))
  nodes = 1000
  layers = 10
  regr = MLPRegressor(hidden_layer_sizes=tuple([nodes] * 4), activation=&quot;relu&quot;).fit(f(X_train_), y_train)
  predicted_sin = regr.predict(f(X_train_))
  
  plt.plot(X_train_, real_sin, label=&quot;sin target&quot;)
  plt.plot(X_train_, predicted_sin, label=&quot;sin predicted&quot;)
  plt.legend()
  plt.show()
  </code></pre>
  <p>You'll see it's not a task too hard to learn:</p>
  <p><a href="https://i.sstatic.net/5iVv7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/5iVv7.png" alt="enter image description here" /></a></p>
  <p>PS: of course this is just of a toy example, and if you decrease layers and amount of hidden units the results will become crap, but it still proves that activation surely affects, but not constrain the non linearity of the final function learned by a neural network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-29T06:21:41.800Z
Answers.Text: I would disagree, that every loss function is non-convex. Mean squared error and  Crossentropy (Binary or Multilabel) are convex functions.
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-29T16:54:17.450Z
Answers.Text: Convex in their formulation, but when applied in neural networks they lose this property, for details check this thread https://math.stackexchange.com/questions/2402455/convexity-of-mse-in-neural-networks/2402472
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-29T18:37:20.677Z
Answers.Text: one needs to specify for convexity with respect to which variable the function is. I spoke about convexity with respect to the input arguments of the loss function. $\mathrm{MSE} = \frac{1}{2}(y - \hat{y})^2$ is convex with respect to the prediction $y$. But if you are speaking about convexity with respect to the NN weights, the resulting function generally will be non-convex.
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-28T16:05:00.787Z
Answers.LastActivityDate: 2022-01-28T16:23:24.873Z
Answers.LastEditDate: 2022-01-28T16:23:24.873Z
Answers.UserDisplayName: Edoardo Guerriero
Answers.Body: <p>I think you are asking about Fully Input Convex Neural Networks as proposed in [1].
  ReLU is in fact a convex function, and the sum of convex functions can only produce convex functions. However, unlike you said, composition of convex functions can produce non-convex functions, unless they are non-decreasing.
  With FICNNs you can only learn convex functions. For that all weights W must be non-negative for the activation function g</p>
  <p>From [1] the interesting part is:</p>
  <blockquote>
  <p>The function f is convex in y provided that
  all <span class="math-container">$W_i(z)_{1:k}$</span> are non-negative, and all functions <span class="math-container">$g_i$</span> are convex and non-decreasing.
  The proof is simple and follows from the fact that non-negative sums of convex functions are also convex and that
  the composition of a convex and convex non-decreasing
  function is also convex (see e.g. Boyd &amp; Vandenberghe
  (2004, 3.2.4)).</p>
  </blockquote>
  <p>[1] Amos, Brandon, Lei Xu, and J. Zico Kolter. &quot;Input convex neural networks.&quot; International Conference on Machine Learning. PMLR, 2017.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-16T14:59:17.823Z
Answers.Text: I'd recommend using LaTeX to format your equations and symbols for example $W_i$ and $g_i$
Answers.UserDisplayName: Saurav Maheshkar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-16T04:24:47.750Z
Answers.LastActivityDate: 2022-11-17T15:18:28.423Z
Answers.LastEditDate: 2022-11-17T15:18:28.423Z
Answers.UserDisplayName: ufg
Body: <p>I have heard the following argument being made regarding Neural Networks:</p>
  <ul>
  <li>A Neural Network is a composition of several Activation Functions</li>
  <li>Sigmoid Activation Functions are Non-Convex Functions</li>
  <li>The composition of Non-Convex Functions can produce a Non-Convex Function</li>
  <li>Thus, Loss Functions for Neural Networks that contain several Sigmoid Activation Functions can be Non-Convex</li>
  </ul>
  <p>Using the R programming language, I plotted the second derivative of the Sigmoid Function and we can see that it fails the Convexity Test (i.e. the second derivative can take both positive and negative values):</p>
  <pre><code>e = 2.718
  
  eq = function(x){ (-e^-x)* (1+e^-x)^-2  + (e^-x)*(-2*(1+e^-x)^-3 *(-e^-x))}
  
  plot(eq(-100:100), type='l', main = &quot;Plot of Second Derivative of the Sigmoid Function&quot;)
  </code></pre>
  <p><a href="https://i.sstatic.net/LGeum.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/LGeum.png" alt="enter image description here" /></a></p>
  <p><strong>My Question:</strong> (If the above argument is in fact true) Can the same argument be extended to lack of Convexity of Loss Functions of Neural Networks containing several &quot;RELU Activation Functions&quot; ?</p>
  <ul>
  <li>On it's own, the ReLU function is said to be Convex.</li>
  <li>Mathematically, we can show that compositions of Convex Functions can only produce a Convex Function.</li>
  </ul>
  <p>However, Neural Networks that contain compositions of (only) ReLU Activation functions make it <strong>unclear to me how a Loss Functions that contains (only) &quot;RELU Activation Functions&quot; would a Non-Convex.</strong></p>
  <p><a href="https://i.sstatic.net/0U2jb.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0U2jb.png" alt="enter image description here" /></a></p>
  <p>Can someone please comment on this? <strong>If compositions of Convex Functions can only produce Convex Functions - does this mean that the Loss Function of a Neural Network containing only containing ReLU Activation Functions can never be Non-Convex?</strong></p>
  <p>Thanks!</p>
  <ul>
  <li><strong>References:</strong></li>
  </ul>
  <p><a href="https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html" rel="nofollow noreferrer">https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html</a></p>
  <p><strong>Note:</strong> Using some informal logic, I do not think that the Loss Functions of Neural Networks containing RELU Activation Functions are generally Convex. This is because RELU (style) Activation Functions are generally some of the most common types of activation functions being used - yet the same difficulties concerning mon-convex optimization still remain. Thus, I would like to think that Neural Networks with RELU Activation Functions are still generally non-convex.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-24T15:24:25.350Z
Comments.Text: single layer with relu activation should be convex. But compositions of convex functions are not necessarily convex, so even in the case of a single layer with relu, the loss function may not be convex.
Comments.UserDisplayName: Taw
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-24T15:26:12.933Z
Comments.Text: Also, you should note that relu is a nonlinear function - just like sigmoid.
Comments.UserDisplayName: Taw
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-24T22:20:02.437Z
Comments.Text: @ Taw: thank you for your reply! what do you mean by "compositions"? do you mean "a loss function composed of several relu functions"?
Comments.UserDisplayName: stats_noob
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-24T22:20:59.393Z
Comments.Text: @ Taw: can you please explain how "Relu is a non-linear function"? it seems to be a piecewise linear function, no?
Comments.UserDisplayName: stats_noob
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-25T01:27:37.850Z
Comments.Text: @stats555 each piece itself is linear, but the function as a whole is non-linear.
Comments.UserDisplayName: SpiderRico
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-25T05:26:25.717Z
Comments.Text: @stats555 piecewise linear functions are not linear, hence they are nonlinear. A composition of two functions $f, y$ is defined as $f(y(x))$ where $x$ is input. E.g. call the model y, call the loss function $f$. Then the loss function used in deep learning would look something like $f(\hat y, y(x))$ where $\hat y$ is the target, $y$ is the model, $x$ is the input. My point is, even if $y$ is a single layer with relu, $f(\hat y, y(x))$ is not necessarily convex.
Comments.UserDisplayName: Taw
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-25T05:31:47.147Z
Comments.Text: @ Taw: thank you so much for your reply! I spent the whole day reading more about this - I think I finally understand what you mentioned in the first part of the reply. I am just wondering about the last part : "even if y is a single layer with relu f(y_hat, y(x)) is not necessarily convex" : is there some way we have of knowing why this is true?
Comments.UserDisplayName: stats_noob
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-25T05:33:54.273Z
Comments.Text: For instance, I tried writing the loss function for a small neural network containing RELU activations. I then tried to evaluate the Hessian Matrix for this loss function. I was able to fill the Hessian Matrix with the second derivatives with respective to each weight from the loss function - but I am still not sure how we can show that this loss function containing RELU is not necessarily convex. Do you have any ideas about this? thank you so much!
Comments.UserDisplayName: stats_noob
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-25T09:42:32.900Z
Comments.Text: Please, avoid cross-posting the same question. Decide which site you want to post and initially post there. Then, if you don't get an answer there, delete the question from the site where you asked, and ask it on another similar/related site.
Comments.UserDisplayName: nbro
Tags: <neural-networks><optimization><relu>
Title: Effects of ReLU Activation on Convexity of Loss Functions
UserDisplayName: stats_noob
ThreadText: # Effects of ReLU Activation on Convexity of Loss Functions
  
  (Asked by: username_0 on 2022-01-24)
  
  I have heard the following argument being made regarding Neural Networks:
  
  * A Neural Network is a composition of several Activation Functions
  * Sigmoid Activation Functions are Non-Convex Functions
  * The composition of Non-Convex Functions can produce a Non-Convex Function
  * Thus, Loss Functions for Neural Networks that contain several Sigmoid Activation Functions can be Non-Convex
  
  Using the R programming language, I plotted the second derivative of the Sigmoid Function and we can see that it fails the Convexity Test (i.e. the second derivative can take both positive and negative values):
  
  ```
  e = 2.718
  
  eq = function(x){ (-e^-x)* (1+e^-x)^-2  + (e^-x)*(-2*(1+e^-x)^-3 *(-e^-x))}
  
  plot(eq(-100:100), type='l', main = "Plot of Second Derivative of the Sigmoid Function")
  
  ```
  
  [![enter image description here](https://i.sstatic.net/LGeum.png)](https://i.sstatic.net/LGeum.png)
  
  **My Question:** (If the above argument is in fact true) Can the same argument be extended to lack of Convexity of Loss Functions of Neural Networks containing several "RELU Activation Functions" ?
  
  * On it's own, the ReLU function is said to be Convex.
  * Mathematically, we can show that compositions of Convex Functions can only produce a Convex Function.
  
  However, Neural Networks that contain compositions of (only) ReLU Activation functions make it **unclear to me how a Loss Functions that contains (only) "RELU Activation Functions" would a Non-Convex.**
  
  [![enter image description here](https://i.sstatic.net/0U2jb.png)](https://i.sstatic.net/0U2jb.png)
  
  Can someone please comment on this? **If compositions of Convex Functions can only produce Convex Functions - does this mean that the Loss Function of a Neural Network containing only containing ReLU Activation Functions can never be Non-Convex?**
  
  Thanks!
  
  * **References:**
  
  <https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html>
  
  **Note:** Using some informal logic, I do not think that the Loss Functions of Neural Networks containing RELU Activation Functions are generally Convex. This is because RELU (style) Activation Functions are generally some of the most common types of activation functions being used - yet the same difficulties concerning mon-convex optimization still remain. Thus, I would like to think that Neural Networks with RELU Activation Functions are still generally non-convex.
  
  **Tags:** neural-networks, optimization, relu
  
  > **Comments:**
  > **username_1**: single layer with relu activation should be convex. But compositions of convex functions are not necessarily convex, so even in the case of a single layer with relu, the loss function may not be convex.  
  > **username_1**: Also, you should note that relu is a nonlinear function - just like sigmoid.  
  > **username_0**: @ Taw: thank you for your reply! what do you mean by "compositions"? do you mean "a loss function composed of several relu functions"?  
  > **username_0**: @ Taw: can you please explain how "Relu is a non-linear function"? it seems to be a piecewise linear function, no?  
  > **username_2**: @stats555 each piece itself is linear, but the function as a whole is non-linear.  
  > **username_1**: @stats555 piecewise linear functions are not linear, hence they are nonlinear. A composition of two functions $f, y$ is defined as $f(y(x))$ where $x$ is input. E.g. call the model y, call the loss function $f$. Then the loss function used in deep learning would look something like $f(\hat y, y(x))$ where $\hat y$ is the target, $y$ is the model, $x$ is the input. My point is, even if $y$ is a single layer with relu, $f(\hat y, y(x))$ is not necessarily convex.  
  > **username_0**: @ Taw: thank you so much for your reply! I spent the whole day reading more about this - I think I finally understand what you mentioned in the first part of the reply. I am just wondering about the last part : "even if y is a single layer with relu f(y_hat, y(x)) is not necessarily convex" : is there some way we have of knowing why this is true?  
  > **username_0**: For instance, I tried writing the loss function for a small neural network containing RELU activations. I then tried to evaluate the Hessian Matrix for this loss function. I was able to fill the Hessian Matrix with the second derivatives with respective to each weight from the loss function - but I am still not sure how we can show that this loss function containing RELU is not necessarily convex. Do you have any ideas about this? thank you so much!  
  > **username_3**: Please, avoid cross-posting the same question. Decide which site you want to post and initially post there. Then, if you don't get an answer there, delete the question from the site where you asked, and ask it on another similar/related site.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2022-01-28. Score: 2
  
  You're missing a couple of quite important concepts:
  
  * [Universal approximation theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem): with enough parameters a neural network can approximate any function.
  * Basically every loss function is non convex. (There is this little problem in machine learning call local minima about which we like to complain a lot :) )
  
  But no need to trust me, just run a simple experiment and try yourself to approximate a non convex function, like **$sin(x)$** with relu:
  
  ```
  from sklearn.neural_network import MLPRegressor
  import numpy as np
  import matplotlib.pyplot as plt
  
  f = lambda x: [[x_] for x_ in x]
  noise_level = 0.1
  X_train_ = np.arange(0, 10, 0.2)
  real_sin = np.sin(X_train_)
  y_train = real_sin + np.random.normal(0, noise_level, len(X_train_))
  nodes = 1000
  layers = 10
  regr = MLPRegressor(hidden_layer_sizes=tuple([nodes] * 4), activation="relu").fit(f(X_train_), y_train)
  predicted_sin = regr.predict(f(X_train_))
  
  plt.plot(X_train_, real_sin, label="sin target")
  plt.plot(X_train_, predicted_sin, label="sin predicted")
  plt.legend()
  plt.show()
  
  ```
  
  You'll see it's not a task too hard to learn:
  
  [![enter image description here](https://i.sstatic.net/5iVv7.png)](https://i.sstatic.net/5iVv7.png)
  
  PS: of course this is just of a toy example, and if you decrease layers and amount of hidden units the results will become crap, but it still proves that activation surely affects, but not constrain the non linearity of the final function learned by a neural network.
  
  > **Comments:**
  > **username_5**: I would disagree, that every loss function is non-convex. Mean squared error and  Crossentropy (Binary or Multilabel) are convex functions.  
  > **username_4**: Convex in their formulation, but when applied in neural networks they lose this property, for details check this thread https://math.stackexchange.com/questions/2402455/convexity-of-mse-in-neural-networks/2402472  
  > **username_5**: one needs to specify for convexity with respect to which variable the function is. I spoke about convexity with respect to the input arguments of the loss function. $\mathrm{MSE} = \frac{1}{2}(y - \hat{y})^2$ is convex with respect to the prediction $y$. But if you are speaking about convexity with respect to the NN weights, the resulting function generally will be non-convex.  
  
  ---
  
  ### Answer by username_6 on 2022-11-16. Score: 1
  
  I think you are asking about Fully Input Convex Neural Networks as proposed in [1].
  ReLU is in fact a convex function, and the sum of convex functions can only produce convex functions. However, unlike you said, composition of convex functions can produce non-convex functions, unless they are non-decreasing.
  With FICNNs you can only learn convex functions. For that all weights W must be non-negative for the activation function g
  
  From [1] the interesting part is:
  
  > The function f is convex in y provided that
  > all $W\_i(z)\_{1:k}$ are non-negative, and all functions $g\_i$ are convex and non-decreasing.
  > The proof is simple and follows from the fact that non-negative sums of convex functions are also convex and that
  > the composition of a convex and convex non-decreasing
  > function is also convex (see e.g. Boyd & Vandenberghe
  > (2004, 3.2.4)).
  
  [1] Amos, Brandon, Lei Xu, and J. Zico Kolter. "Input convex neural networks." International Conference on Machine Learning. PMLR, 2017.
  
  > **Comments:**
  > **username_7**: I'd recommend using LaTeX to format your equations and symbols for example $W_i$ and $g_i$  
  
  ---
  
  