<p>The key is that the layers of neurons in neural networks are <em>not</em> affine transformations. All commonly used neurons have some kind of non-linearity. The simplest of these is the Rectified Linear Unit (ReLU), which takes the form <span class="math-container">$y = x$</span> when <span class="math-container">$x &gt; 0$</span> and <span class="math-container">$y = 0$</span> for all other values, where <span class="math-container">$x$</span> is a weighted sum of the inputs to the neuron.</p>

CC BY-SA 4.0
2019-10-10T03:44:09.560Z
I'm wondering if he read somewhere that linear networks perform better than the reduced architecture. Perhaps someone did an experiment with a linear network and its reduced equivalent and discovered that in practice the network performs better than the reduced form. I asked  OP to clarify.
respectful
CC BY-SA 4.0
2019-10-10T13:01:13.977Z
@respectful Maybe. I think a lot of modern presentations show a linear algebraic presentation of neutral networks, and that for a new reader, it can be easy to miss the activation functions and their roles.
John Doucette
CC BY-SA 4.0
2019-10-10T01:08:56.700Z
2019-10-10T01:08:56.700Z
John Doucette
<p>It is a well-known math fact that composition of linear/affine transformations is still linear/affine. For a naive example,</p>
<p><span class="math-container">$\textbf{A}_1\textbf{A}_2\textbf{x}$</span> is simply <span class="math-container">$\textbf{A}\textbf{x}$</span> where <span class="math-container">$\textbf{A}=\textbf{A}_1\textbf{A}_2$</span></p>
<p>Any one knows why in practice multiple linear layers tend to work better, even though it is mathematically equivalent to a single linear layer? Any reference is appreciated!</p>

CC BY-SA 4.0
2019-10-10T01:54:00.347Z
Are you sure those "multiple linear layers" don't have nonlinear activations between them?
user76284
CC BY-SA 4.0
2019-10-10T03:41:57.017Z
can you post a link to a source that explains the result you mention. There is some confusion as to whether or not you mean that a linear network performs better than its "mathematically reduced" network or if you misunderstand network activations. So a link to a source would be helpful.
respectful
<neural-networks>
Any explanation why multiple linear layers work better than a single linear layer in practice?
Tony
# Any explanation why multiple linear layers work better than a single linear layer in practice?

(Asked by: username_0 on 2019-10-09)

It is a well-known math fact that composition of linear/affine transformations is still linear/affine. For a naive example,

$\textbf{A}\_1\textbf{A}\_2\textbf{x}$ is simply $\textbf{A}\textbf{x}$ where $\textbf{A}=\textbf{A}\_1\textbf{A}\_2$

Any one knows why in practice multiple linear layers tend to work better, even though it is mathematically equivalent to a single linear layer? Any reference is appreciated!

**Tags:** neural-networks

> **Comments:**
> **username_1**: Are you sure those "multiple linear layers" don't have nonlinear activations between them?  
> **username_2**: can you post a link to a source that explains the result you mention. There is some confusion as to whether or not you mean that a linear network performs better than its "mathematically reduced" network or if you misunderstand network activations. So a link to a source would be helpful.  

---

## 1 answers

### Answer by username_3 on 2019-10-10. Score: 2

The key is that the layers of neurons in neural networks are *not* affine transformations. All commonly used neurons have some kind of non-linearity. The simplest of these is the Rectified Linear Unit (ReLU), which takes the form $y = x$ when $x > 0$ and $y = 0$ for all other values, where $x$ is a weighted sum of the inputs to the neuron.

> **Comments:**
> **username_2**: I'm wondering if he read somewhere that linear networks perform better than the reduced architecture. Perhaps someone did an experiment with a linear network and its reduced equivalent and discovered that in practice the network performs better than the reduced form. I asked  OP to clarify.  
> **username_3**: @username_2 Maybe. I think a lot of modern presentations show a linear algebraic presentation of neutral networks, and that for a new reader, it can be easy to miss the activation functions and their roles.  

---

