Answers.Body: <blockquote>
  <p>However I when thinking graphically I think that it is possible to approximate these nonlinear functions using lots of linear ( scaled and shifted) linear lines and I do not understand why this is incorrect.</p>
  </blockquote>
  <p>It's not generally incorrect, but what you are describing there is not a single function but a <a href="https://www.math.net/piecewise-functions" rel="nofollow noreferrer"><strong>piecewise function</strong></a>. For example:</p>
  <p><span class="math-container">$
  f(x) = \left\{
          \begin{array}{ll}
              0.1 x &amp; \quad 0 \leq x &lt; 0.1 \\
              0.2 x &amp; \quad 0.1 \leq x &lt; 0.2\\
              0.3 x &amp; \quad ...
          \end{array}
      \right.
  $</span></p>
  <p>This composition is non-linear, because the input space gets transformed differently depending on <span class="math-container">$x$</span> (because each subfunction in the example above has a different slope). There are methods to optimize such a function as well, e.g. <a href="https://medium.com/databulls/piecewise-linear-regression-30b63ff8a90d" rel="nofollow noreferrer">piecewise linear regression</a>.</p>
  <p>An MLP without non-linearities is not a piecewise function, but a <a href="https://en.wikipedia.org/wiki/Function_composition" rel="nofollow noreferrer"><strong>function composition</strong></a>:</p>
  <p><span class="math-container">$(f \circ g)(x) = f(g(x))$</span>,</p>
  <p>where <span class="math-container">$g$</span> and <span class="math-container">$f$</span> are two consecutive dense layers.
  The difference is that if both functions <span class="math-container">$g$</span> and <span class="math-container">$f$</span> are linear, then <span class="math-container">$f \circ g$</span> is linear. This holds for any configuration of model parameters. Therefore, the model transforms any given input space <span class="math-container">$X$</span> into some output space <span class="math-container">$W$</span>, where straight lines in the input space remain straight in the output space and can thus never model something non-linear.</p>
  <blockquote>
  <p>In case, if it is possible to approximate any function using Relu activation function ( which is linear on first quadrant) , why it is not possible to approximate with a function completely linear ?</p>
  </blockquote>
  <p><strong>As soon as you introduce any non-linearity</strong>, the system becomes much more expressive, because each neuron now models a non-linear function. Combining the neurons (as done by a subsequent layer) can learn combinations of these non-linear functions.</p>
  <p>And this is where the two worlds collide: If you take a <code>ReLU</code> activation, it allows the model to actually learn something like a piecewise linear function, because of the combinations of several linear functions with different slopes that are <span class="math-container">$&lt; 0$</span> only for certain input ranges. Here is a simple example of that:</p>
  <pre><code>import numpy as np
  import tensorflow as tf
  import matplotlib.pyplot as plt
  
  tf.random.set_seed(4)
  np.random.seed(4)
  
  # create a dataset
  xs = np.linspace(-2, 2, 1000)
  ys = xs**2 + 0.2
  
  # create a model
  m = tf.keras.Sequential([
      tf.keras.layers.Dense(4, activation='relu'),
      tf.keras.layers.Dense(1),
  ])
  
  # train the model
  m.compile(optimizer='adam', loss=tf.keras.losses.MAE)
  m.fit(xs[:,np.newaxis], ys[:,np.newaxis], batch_size=8, epochs=20, verbose=None)
  
  # plot the results
  fig, ax = plt.subplots(1, 2, figsize=(15, 6))
  
  ax[0].plot(xs, xs, label='model inputs')
  ax[0].plot(xs, ys, label='ground truth')
  ax[0].plot(xs, m(xs)[:,0], label='prediction')
  ax[1].plot(xs, ys)
  
  for n in range(m.layers[0].units):
      neuron_act = m.layers[0](xs[:,np.newaxis])[:,n]
      ax[1].plot(xs, neuron_act, color='g', label='piecewise function %d' % n)
                 
  ax[0].legend()
  ax[1].legend()
  ax[0].set_ylim((-1, 4))
  ax[1].set_ylim((-1, 4))
  
  </code></pre>
  <p><a href="https://i.sstatic.net/FTEDH.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/FTEDH.png" alt="enter image description here" /></a></p>
  <p>The left image shows what the model has learned (green line), the right side shows the functions of the individual neurons. The more neurons you add to that first <code>relu</code>-layer, the better the approximation of the ground truth will be.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-09T13:52:47.603Z
Answers.Text: Thank you very much, great explanation and demonstration. The point is that using linear functions ( exactly, not piece-wise) and combining them in some way, we can never obtain the example function you gave right ?
Answers.UserDisplayName: levitatmas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-09T13:58:00.217Z
Answers.Text: Right! To achieve that, we would have to partition the function which itself is a non-linearity :)
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-09T13:23:38.787Z
Answers.LastActivityDate: 2022-07-09T13:31:07.727Z
Answers.LastEditDate: 2022-07-09T13:31:07.727Z
Answers.UserDisplayName: Chillston
Body: <p>In neural networks we use nonlinear activation functions such as sigmoid, <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)" rel="nofollow noreferrer">ReLU</a>, etc. Using a combination of these functions (with required scaling and shifting), we manage to estimate any nonlinear function.</p>
  <p>I understand in theoretically that we cannot obtain a function such as e<sup>x<sub>1</sub> </sup> + x<sub>2</sub><sup>2</sup> ...(and other such nonlinear components of inputs x<sub>3</sub>, x<sub>4</sub>, etc.) using only linear combinations of x<sub>i</sub>'s. However, when I am thinking graphically I think that it is possible to approximate these nonlinear functions using lots of linear (scaled and shifted) linear lines and I do not understand why this is incorrect.</p>
  <p>In case, if it is possible to approximate any function using a ReLU activation function (which is linear in the first quadrant), why is it not possible to approximate with a function completely linear?</p>
  
Tags: <neural-networks><machine-learning><activation-functions><relu>
Title: Why cannot linear activation functions be used to approximate any function?
UserDisplayName: levitatmas
ThreadText: # Why cannot linear activation functions be used to approximate any function?
  
  (Asked by: username_0 on 2022-07-09)
  
  In neural networks we use nonlinear activation functions such as sigmoid, [ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)), etc. Using a combination of these functions (with required scaling and shifting), we manage to estimate any nonlinear function.
  
  I understand in theoretically that we cannot obtain a function such as ex1  + x22 ...(and other such nonlinear components of inputs x3, x4, etc.) using only linear combinations of xi's. However, when I am thinking graphically I think that it is possible to approximate these nonlinear functions using lots of linear (scaled and shifted) linear lines and I do not understand why this is incorrect.
  
  In case, if it is possible to approximate any function using a ReLU activation function (which is linear in the first quadrant), why is it not possible to approximate with a function completely linear?
  
  **Tags:** neural-networks, machine-learning, activation-functions, relu
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-07-09. Score: 3 | Accepted answer
  
  However I when thinking graphically I think that it is possible to approximate these nonlinear functions using lots of linear ( scaled and shifted) linear lines and I do not understand why this is incorrect.
  
  It's not generally incorrect, but what you are describing there is not a single function but a piecewise function. For example:
  $
  f(x) = \left\{
          \begin{array}{ll}
              0.1 x & \quad 0 \leq x < 0.1 \\
              0.2 x & \quad 0.1 \leq x < 0.2\\
              0.3 x & \quad ...
          \end{array}
      \right.
  $
  This composition is non-linear, because the input space gets transformed differently depending on $x$ (because each subfunction in the example above has a different slope). There are methods to optimize such a function as well, e.g. piecewise linear regression.
  An MLP without non-linearities is not a piecewise function, but a function composition:
  $(f \circ g)(x) = f(g(x))$,
  where $g$ and $f$ are two consecutive dense layers.
  The difference is that if both functions $g$ and $f$ are linear, then $f \circ g$ is linear. This holds for any configuration of model parameters. Therefore, the model transforms any given input space $X$ into some output space $W$, where straight lines in the input space remain straight in the output space and can thus never model something non-linear.
  
  In case, if it is possible to approximate any function using Relu activation function ( which is linear on first quadrant) , why it is not possible to approximate with a function completely linear ?
  
  As soon as you introduce any non-linearity, the system becomes much more expressive, because each neuron now models a non-linear function. Combining the neurons (as done by a subsequent layer) can learn combinations of these non-linear functions.
  And this is where the two worlds collide: If you take a ReLU activation, it allows the model to actually learn something like a piecewise linear function, because of the combinations of several linear functions with different slopes that are $< 0$ only for certain input ranges. Here is a simple example of that:
  import numpy as np
  import tensorflow as tf
  import matplotlib.pyplot as plt
  
  tf.random.set_seed(4)
  np.random.seed(4)
  
  # create a dataset
  xs = np.linspace(-2, 2, 1000)
  ys = xs**2 + 0.2
  
  # create a model
  m = tf.keras.Sequential([
      tf.keras.layers.Dense(4, activation='relu'),
      tf.keras.layers.Dense(1),
  ])
  
  # train the model
  m.compile(optimizer='adam', loss=tf.keras.losses.MAE)
  m.fit(xs[:,np.newaxis], ys[:,np.newaxis], batch_size=8, epochs=20, verbose=None)
  
  # plot the results
  fig, ax = plt.subplots(1, 2, figsize=(15, 6))
  
  ax[0].plot(xs, xs, label='model inputs')
  ax[0].plot(xs, ys, label='ground truth')
  ax[0].plot(xs, m(xs)[:,0], label='prediction')
  ax[1].plot(xs, ys)
  
  for n in range(m.layers[0].units):
      neuron_act = m.layers[0](xs[:,np.newaxis])[:,n]
      ax[1].plot(xs, neuron_act, color='g', label='piecewise function %d' % n)
                 
  ax[0].legend()
  ax[1].legend()
  ax[0].set_ylim((-1, 4))
  ax[1].set_ylim((-1, 4))
  
  
  
  The left image shows what the model has learned (green line), the right side shows the functions of the individual neurons. The more neurons you add to that first relu-layer, the better the approximation of the ground truth will be.
  
  > **Comments:**
  > **username_0**: Thank you very much, great explanation and demonstration. The point is that using linear functions ( exactly, not piece-wise) and combining them in some way, we can never obtain the example function you gave right ?  
  > **username_1**: Right! To achieve that, we would have to partition the function which itself is a non-linearity :)  
  
  ---
  
  