Answers.Body: <p>The concept of non-linearity is not only restricted to Convolutional Networks but can be seen in RNNs, and any feed forward neural networks.</p>
  <p>Without a non-linear activation function, two feed forward layers can be collapsed into a single feed forward layer. So, let's say you have a large neural network with 5 layers possessing weights W1 to W5.</p>
  <p><strong>Without a non-linear layer</strong>, passing an input x would give you the following:</p>
  <p><strong>y = W5(W4(W3(W2(W1(x)))))</strong>
  which is equivalent to <strong>y = W(x)</strong> where <strong>W is a product of W1..W5</strong></p>
  <p>which is essentially a single linear layer -Hence, this would not allow the neural network to learn higher order or more complicated functions which are not linear in nature.</p>
  <p>The presence of a non-linearity between the layers allows the weights to be adjusted in order to learn complex functions apart from linear functions. Eg. if your data might be quadratic in nature, or say some complicated mixture of logarithmic, sinusoidal functions, hyperbolic functions, etc.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-01T09:09:48.533Z
Answers.Text: You can use mathjax on this site, so I recommend that you do that.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-01T03:28:26.127Z
Answers.LastActivityDate: 2022-03-01T03:28:26.127Z
Answers.UserDisplayName: Caxton
Body: <p>I'm aware of the working of ReLU that it's turns every negative value to zero and doesn't effect any positive value, but what confuses me is that: what is actually meant by Non-linearity in feature map after Convolution operation on any image? What actually is non-linearity? And why do we focus on removing it completely from the feature map?</p>
  <p>Thank you.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-01T09:16:36.817Z
Comments.Text: Who said that "we focus on removing it completely from the feature map"?
Comments.UserDisplayName: nbro
Tags: <deep-learning><convolutional-neural-networks><image-processing><relu>
Title: What is meant by non-linearity in Convolutional Neural Networks? And why do we focus on removing it entirely?
UserDisplayName: Sharjeel M. Rajput
ThreadText: # What is meant by non-linearity in Convolutional Neural Networks? And why do we focus on removing it entirely?
  
  (Asked by: username_0 on 2022-02-28)
  
  I'm aware of the working of ReLU that it's turns every negative value to zero and doesn't effect any positive value, but what confuses me is that: what is actually meant by Non-linearity in feature map after Convolution operation on any image? What actually is non-linearity? And why do we focus on removing it completely from the feature map?
  
  Thank you.
  
  **Tags:** deep-learning, convolutional-neural-networks, image-processing, relu
  
  > **Comments:**
  > **username_1**: Who said that "we focus on removing it completely from the feature map"?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-03-01. Score: 1 | Accepted answer
  
  The concept of non-linearity is not only restricted to Convolutional Networks but can be seen in RNNs, and any feed forward neural networks.
  
  Without a non-linear activation function, two feed forward layers can be collapsed into a single feed forward layer. So, let's say you have a large neural network with 5 layers possessing weights W1 to W5.
  
  **Without a non-linear layer**, passing an input x would give you the following:
  
  **y = W5(W4(W3(W2(W1(x)))))**
  which is equivalent to **y = W(x)** where **W is a product of W1..W5**
  
  which is essentially a single linear layer -Hence, this would not allow the neural network to learn higher order or more complicated functions which are not linear in nature.
  
  The presence of a non-linearity between the layers allows the weights to be adjusted in order to learn complex functions apart from linear functions. Eg. if your data might be quadratic in nature, or say some complicated mixture of logarithmic, sinusoidal functions, hyperbolic functions, etc.
  
  > **Comments:**
  > **username_1**: You can use mathjax on this site, so I recommend that you do that.  
  
  ---
  
  