<blockquote>
<p>However I when thinking graphically I think that it is possible to approximate these nonlinear functions using lots of linear ( scaled and shifted) linear lines and I do not understand why this is incorrect.</p>
</blockquote>
<p>It's not generally incorrect, but what you are describing there is not a single function but a <a href="https://www.math.net/piecewise-functions" rel="nofollow noreferrer"><strong>piecewise function</strong></a>. For example:</p>
<p><span class="math-container">$
f(x) = \left\{
        \begin{array}{ll}
            0.1 x &amp; \quad 0 \leq x &lt; 0.1 \\
            0.2 x &amp; \quad 0.1 \leq x &lt; 0.2\\
            0.3 x &amp; \quad ...
        \end{array}
    \right.
$</span></p>
<p>This composition is non-linear, because the input space gets transformed differently depending on <span class="math-container">$x$</span> (because each subfunction in the example above has a different slope). There are methods to optimize such a function as well, e.g. <a href="https://medium.com/databulls/piecewise-linear-regression-30b63ff8a90d" rel="nofollow noreferrer">piecewise linear regression</a>.</p>
<p>An MLP without non-linearities is not a piecewise function, but a <a href="https://en.wikipedia.org/wiki/Function_composition" rel="nofollow noreferrer"><strong>function composition</strong></a>:</p>
<p><span class="math-container">$(f \circ g)(x) = f(g(x))$</span>,</p>
<p>where <span class="math-container">$g$</span> and <span class="math-container">$f$</span> are two consecutive dense layers.
The difference is that if both functions <span class="math-container">$g$</span> and <span class="math-container">$f$</span> are linear, then <span class="math-container">$f \circ g$</span> is linear. This holds for any configuration of model parameters. Therefore, the model transforms any given input space <span class="math-container">$X$</span> into some output space <span class="math-container">$W$</span>, where straight lines in the input space remain straight in the output space and can thus never model something non-linear.</p>
<blockquote>
<p>In case, if it is possible to approximate any function using Relu activation function ( which is linear on first quadrant) , why it is not possible to approximate with a function completely linear ?</p>
</blockquote>
<p><strong>As soon as you introduce any non-linearity</strong>, the system becomes much more expressive, because each neuron now models a non-linear function. Combining the neurons (as done by a subsequent layer) can learn combinations of these non-linear functions.</p>
<p>And this is where the two worlds collide: If you take a <code>ReLU</code> activation, it allows the model to actually learn something like a piecewise linear function, because of the combinations of several linear functions with different slopes that are <span class="math-container">$&lt; 0$</span> only for certain input ranges. Here is a simple example of that:</p>
<pre><code>import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt

tf.random.set_seed(4)
np.random.seed(4)

# create a dataset
xs = np.linspace(-2, 2, 1000)
ys = xs**2 + 0.2

# create a model
m = tf.keras.Sequential([
    tf.keras.layers.Dense(4, activation='relu'),
    tf.keras.layers.Dense(1),
])

# train the model
m.compile(optimizer='adam', loss=tf.keras.losses.MAE)
m.fit(xs[:,np.newaxis], ys[:,np.newaxis], batch_size=8, epochs=20, verbose=None)

# plot the results
fig, ax = plt.subplots(1, 2, figsize=(15, 6))

ax[0].plot(xs, xs, label='model inputs')
ax[0].plot(xs, ys, label='ground truth')
ax[0].plot(xs, m(xs)[:,0], label='prediction')
ax[1].plot(xs, ys)

for n in range(m.layers[0].units):
    neuron_act = m.layers[0](xs[:,np.newaxis])[:,n]
    ax[1].plot(xs, neuron_act, color='g', label='piecewise function %d' % n)
               
ax[0].legend()
ax[1].legend()
ax[0].set_ylim((-1, 4))
ax[1].set_ylim((-1, 4))

</code></pre>
<p><a href="https://i.sstatic.net/FTEDH.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/FTEDH.png" alt="enter image description here" /></a></p>
<p>The left image shows what the model has learned (green line), the right side shows the functions of the individual neurons. The more neurons you add to that first <code>relu</code>-layer, the better the approximation of the ground truth will be.</p>

CC BY-SA 4.0
2022-07-09T13:52:47.603Z
Thank you very much, great explanation and demonstration. The point is that using linear functions ( exactly, not piece-wise) and combining them in some way, we can never obtain the example function you gave right ?
levitatmas
CC BY-SA 4.0
2022-07-09T13:58:00.217Z
Right! To achieve that, we would have to partition the function which itself is a non-linearity :)
Chillston
CC BY-SA 4.0
2022-07-09T13:23:38.787Z
2022-07-09T13:31:07.727Z
2022-07-09T13:31:07.727Z
Chillston
<p>In neural networks we use nonlinear activation functions such as sigmoid, <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)" rel="nofollow noreferrer">ReLU</a>, etc. Using a combination of these functions (with required scaling and shifting), we manage to estimate any nonlinear function.</p>
<p>I understand in theoretically that we cannot obtain a function such as e<sup>x<sub>1</sub> </sup> + x<sub>2</sub><sup>2</sup> ...(and other such nonlinear components of inputs x<sub>3</sub>, x<sub>4</sub>, etc.) using only linear combinations of x<sub>i</sub>'s. However, when I am thinking graphically I think that it is possible to approximate these nonlinear functions using lots of linear (scaled and shifted) linear lines and I do not understand why this is incorrect.</p>
<p>In case, if it is possible to approximate any function using a ReLU activation function (which is linear in the first quadrant), why is it not possible to approximate with a function completely linear?</p>

<neural-networks><machine-learning><activation-functions><relu>
Why cannot linear activation functions be used to approximate any function?
levitatmas
# Why cannot linear activation functions be used to approximate any function?

(Asked by: username_0 on 2022-07-09)

In neural networks we use nonlinear activation functions such as sigmoid, [ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)), etc. Using a combination of these functions (with required scaling and shifting), we manage to estimate any nonlinear function.

I understand in theoretically that we cannot obtain a function such as ex1  + x22 ...(and other such nonlinear components of inputs x3, x4, etc.) using only linear combinations of xi's. However, when I am thinking graphically I think that it is possible to approximate these nonlinear functions using lots of linear (scaled and shifted) linear lines and I do not understand why this is incorrect.

In case, if it is possible to approximate any function using a ReLU activation function (which is linear in the first quadrant), why is it not possible to approximate with a function completely linear?

**Tags:** neural-networks, machine-learning, activation-functions, relu


---

## 1 answers

### Answer by username_1 on 2022-07-09. Score: 3 | Accepted answer

However I when thinking graphically I think that it is possible to approximate these nonlinear functions using lots of linear ( scaled and shifted) linear lines and I do not understand why this is incorrect.

It's not generally incorrect, but what you are describing there is not a single function but a piecewise function. For example:
$
f(x) = \left\{
        \begin{array}{ll}
            0.1 x & \quad 0 \leq x < 0.1 \\
            0.2 x & \quad 0.1 \leq x < 0.2\\
            0.3 x & \quad ...
        \end{array}
    \right.
$
This composition is non-linear, because the input space gets transformed differently depending on $x$ (because each subfunction in the example above has a different slope). There are methods to optimize such a function as well, e.g. piecewise linear regression.
An MLP without non-linearities is not a piecewise function, but a function composition:
$(f \circ g)(x) = f(g(x))$,
where $g$ and $f$ are two consecutive dense layers.
The difference is that if both functions $g$ and $f$ are linear, then $f \circ g$ is linear. This holds for any configuration of model parameters. Therefore, the model transforms any given input space $X$ into some output space $W$, where straight lines in the input space remain straight in the output space and can thus never model something non-linear.

In case, if it is possible to approximate any function using Relu activation function ( which is linear on first quadrant) , why it is not possible to approximate with a function completely linear ?

As soon as you introduce any non-linearity, the system becomes much more expressive, because each neuron now models a non-linear function. Combining the neurons (as done by a subsequent layer) can learn combinations of these non-linear functions.
And this is where the two worlds collide: If you take a ReLU activation, it allows the model to actually learn something like a piecewise linear function, because of the combinations of several linear functions with different slopes that are $< 0$ only for certain input ranges. Here is a simple example of that:
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt

tf.random.set_seed(4)
np.random.seed(4)

# create a dataset
xs = np.linspace(-2, 2, 1000)
ys = xs**2 + 0.2

# create a model
m = tf.keras.Sequential([
    tf.keras.layers.Dense(4, activation='relu'),
    tf.keras.layers.Dense(1),
])

# train the model
m.compile(optimizer='adam', loss=tf.keras.losses.MAE)
m.fit(xs[:,np.newaxis], ys[:,np.newaxis], batch_size=8, epochs=20, verbose=None)

# plot the results
fig, ax = plt.subplots(1, 2, figsize=(15, 6))

ax[0].plot(xs, xs, label='model inputs')
ax[0].plot(xs, ys, label='ground truth')
ax[0].plot(xs, m(xs)[:,0], label='prediction')
ax[1].plot(xs, ys)

for n in range(m.layers[0].units):
    neuron_act = m.layers[0](xs[:,np.newaxis])[:,n]
    ax[1].plot(xs, neuron_act, color='g', label='piecewise function %d' % n)
               
ax[0].legend()
ax[1].legend()
ax[0].set_ylim((-1, 4))
ax[1].set_ylim((-1, 4))



The left image shows what the model has learned (green line), the right side shows the functions of the individual neurons. The more neurons you add to that first relu-layer, the better the approximation of the ground truth will be.

> **Comments:**
> **username_0**: Thank you very much, great explanation and demonstration. The point is that using linear functions ( exactly, not piece-wise) and combining them in some way, we can never obtain the example function you gave right ?  
> **username_1**: Right! To achieve that, we would have to partition the function which itself is a non-linearity :)  

---

