Answers.Body: <p>The key is that the layers of neurons in neural networks are <em>not</em> affine transformations. All commonly used neurons have some kind of non-linearity. The simplest of these is the Rectified Linear Unit (ReLU), which takes the form <span class="math-container">$y = x$</span> when <span class="math-container">$x &gt; 0$</span> and <span class="math-container">$y = 0$</span> for all other values, where <span class="math-container">$x$</span> is a weighted sum of the inputs to the neuron.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-10T03:44:09.560Z
Answers.Text: I'm wondering if he read somewhere that linear networks perform better than the reduced architecture. Perhaps someone did an experiment with a linear network and its reduced equivalent and discovered that in practice the network performs better than the reduced form. I asked  OP to clarify.
Answers.UserDisplayName: respectful
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-10T13:01:13.977Z
Answers.Text: @respectful Maybe. I think a lot of modern presentations show a linear algebraic presentation of neutral networks, and that for a new reader, it can be easy to miss the activation functions and their roles.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-10T01:08:56.700Z
Answers.LastActivityDate: 2019-10-10T01:08:56.700Z
Answers.UserDisplayName: John Doucette
Body: <p>It is a well-known math fact that composition of linear/affine transformations is still linear/affine. For a naive example,</p>
  <p><span class="math-container">$\textbf{A}_1\textbf{A}_2\textbf{x}$</span> is simply <span class="math-container">$\textbf{A}\textbf{x}$</span> where <span class="math-container">$\textbf{A}=\textbf{A}_1\textbf{A}_2$</span></p>
  <p>Any one knows why in practice multiple linear layers tend to work better, even though it is mathematically equivalent to a single linear layer? Any reference is appreciated!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-10T01:54:00.347Z
Comments.Text: Are you sure those "multiple linear layers" don't have nonlinear activations between them?
Comments.UserDisplayName: user76284
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-10T03:41:57.017Z
Comments.Text: can you post a link to a source that explains the result you mention. There is some confusion as to whether or not you mean that a linear network performs better than its "mathematically reduced" network or if you misunderstand network activations. So a link to a source would be helpful.
Comments.UserDisplayName: respectful
Tags: <neural-networks>
Title: Any explanation why multiple linear layers work better than a single linear layer in practice?
UserDisplayName: Tony
ThreadText: # Any explanation why multiple linear layers work better than a single linear layer in practice?
  
  (Asked by: username_0 on 2019-10-09)
  
  It is a well-known math fact that composition of linear/affine transformations is still linear/affine. For a naive example,
  
  $\textbf{A}\_1\textbf{A}\_2\textbf{x}$ is simply $\textbf{A}\textbf{x}$ where $\textbf{A}=\textbf{A}\_1\textbf{A}\_2$
  
  Any one knows why in practice multiple linear layers tend to work better, even though it is mathematically equivalent to a single linear layer? Any reference is appreciated!
  
  **Tags:** neural-networks
  
  > **Comments:**
  > **username_1**: Are you sure those "multiple linear layers" don't have nonlinear activations between them?  
  > **username_2**: can you post a link to a source that explains the result you mention. There is some confusion as to whether or not you mean that a linear network performs better than its "mathematically reduced" network or if you misunderstand network activations. So a link to a source would be helpful.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2019-10-10. Score: 2
  
  The key is that the layers of neurons in neural networks are *not* affine transformations. All commonly used neurons have some kind of non-linearity. The simplest of these is the Rectified Linear Unit (ReLU), which takes the form $y = x$ when $x > 0$ and $y = 0$ for all other values, where $x$ is a weighted sum of the inputs to the neuron.
  
  > **Comments:**
  > **username_2**: I'm wondering if he read somewhere that linear networks perform better than the reduced architecture. Perhaps someone did an experiment with a linear network and its reduced equivalent and discovered that in practice the network performs better than the reduced form. I asked  OP to clarify.  
  > **username_3**: @username_2 Maybe. I think a lot of modern presentations show a linear algebraic presentation of neutral networks, and that for a new reader, it can be easy to miss the activation functions and their roles.  
  
  ---
  
  