Answers.Body: <p>These are just two <em>equivalent</em> interpretations (or illustrations) of the application of an activation function. In other words, in a multi-layer perceptron (MLP), you could also illustrate the application of the activation function as a separate layer that follows a linear combination layer. However, in the context of MLPs, the math is relatively simple and <em>elegant</em>, so a fully-connected layer of an MLP can simply be represented as follows</p>
  
  <p><span class="math-container">$$
  \sigma \left(\mathbf{W} \mathbf{X} + \mathbf{b} \right)
  $$</span></p>
  
  <p>where <span class="math-container">$\sigma$</span> is some activation function and <span class="math-container">$\mathbf{W} \mathbf{X}$</span> is the linear combination of the inputs, <span class="math-container">$\mathbf{X}$</span>, and the weights, <span class="math-container">$\mathbf{W}$</span>, and <span class="math-container">$\mathbf{b}$</span> is a bias. You could even represent a full or complete MLP (and not just one fully-connected layer) as a <a href="https://en.wikipedia.org/wiki/Function_composition" rel="nofollow noreferrer">composite (or nested) function</a>. </p>
  
  <p>In the context of convolutional neural networks (CNNs), people might illustrate the application of the activation function as a separate layer because the application of an activation function to the result of the convolution operation of the CNN is <strong>optional</strong> and <strong>out-of-favor</strong> (as stated in this article <a href="http://cs231n.github.io/convolutional-networks/#norm" rel="nofollow noreferrer">http://cs231n.github.io/convolutional-networks</a>), as opposed to the case of MLPs, where activations functions usually follow the linear combination. However, note that the last layers of a CNN are usually fully-connected layers (and not convolutional or pooling layers), that is, they are a linear combination of their input and their weights followed by an application of an activation function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-20T20:39:50.663Z
Answers.Text: **Advice**: _you should expect different, ambiguous, inconsistent and even contradictory notations and terminologies that are used to describe the same (or similar) concepts_. You need to be able to deal with these unfortunate inconsistencies.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-20T20:25:40.017Z
Answers.LastActivityDate: 2019-07-20T20:36:54.257Z
Answers.LastEditDate: 2019-07-20T20:36:54.257Z
Answers.UserDisplayName: nbro
Body: <p>I have been reading up on CNNs. One of the different confusing things has been that people always talk of normalization <strong>layers</strong>. A common normalization layer is a ReLU layer. But I never encountered an explanation of why all of a sudden, activation functions become their own layers in CNNs, while they are only <strong>parts</strong> of a fully connected layer in MLPs.</p>
  
  <p>What is the reason for having dedicated activation layers in CNNs rather than applying the activation to the output volume of a convolutional layer as part of the convolutional layer, as it is the case for dense layers in MLPs?</p>
  
  <p>I guess, in the end, there is no functional difference. We could just as well have separate activation layers in MLPs rather than activation functions in their fully connected layers. But this difference in the convention is irritating still. Well, assuming it only <strong>is</strong> an artifact of the convention.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-20T20:15:18.900Z
Comments.Text: Hi and welcome to this community! Can you please cite an article that talks about normalisation layers in CNNs?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-20T20:18:22.997Z
Comments.Text: Sure, this one for example: http://cs231n.github.io/convolutional-networks/
Comments.UserDisplayName: lo tolmencre
Tags: <neural-networks><convolutional-neural-networks><definitions><activation-functions><multilayer-perceptrons>
Title: Why are activation functions independent layers in CNNs rather than part of convolutional layers?
UserDisplayName: lo tolmencre
ThreadText: # Why are activation functions independent layers in CNNs rather than part of convolutional layers?
  
  (Asked by: username_0 on 2019-07-20)
  
  I have been reading up on CNNs. One of the different confusing things has been that people always talk of normalization **layers**. A common normalization layer is a ReLU layer. But I never encountered an explanation of why all of a sudden, activation functions become their own layers in CNNs, while they are only **parts** of a fully connected layer in MLPs.
  
  What is the reason for having dedicated activation layers in CNNs rather than applying the activation to the output volume of a convolutional layer as part of the convolutional layer, as it is the case for dense layers in MLPs?
  
  I guess, in the end, there is no functional difference. We could just as well have separate activation layers in MLPs rather than activation functions in their fully connected layers. But this difference in the convention is irritating still. Well, assuming it only **is** an artifact of the convention.
  
  **Tags:** neural-networks, convolutional-neural-networks, definitions, activation-functions, multilayer-perceptrons
  > **Comments:**
  > **username_1**: Hi and welcome to this community! Can you please cite an article that talks about normalisation layers in CNNs?  
  > **username_0**: Sure, this one for example: http://cs231n.github.io/convolutional-networks/  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-07-20. Score: 1 | Accepted answer
  
  These are just two *equivalent* interpretations (or illustrations) of the application of an activation function. In other words, in a multi-layer perceptron (MLP), you could also illustrate the application of the activation function as a separate layer that follows a linear combination layer. However, in the context of MLPs, the math is relatively simple and *elegant*, so a fully-connected layer of an MLP can simply be represented as follows
  
  $$
  \sigma \left(\mathbf{W} \mathbf{X} + \mathbf{b} \right)
  $$
  
  where $\sigma$ is some activation function and $\mathbf{W} \mathbf{X}$ is the linear combination of the inputs, $\mathbf{X}$, and the weights, $\mathbf{W}$, and $\mathbf{b}$ is a bias. You could even represent a full or complete MLP (and not just one fully-connected layer) as a [composite (or nested) function](https://en.wikipedia.org/wiki/Function_composition).
  
  In the context of convolutional neural networks (CNNs), people might illustrate the application of the activation function as a separate layer because the application of an activation function to the result of the convolution operation of the CNN is **optional** and **out-of-favor** (as stated in this article [http://cs231n.github.io/convolutional-networks](http://cs231n.github.io/convolutional-networks/#norm)), as opposed to the case of MLPs, where activations functions usually follow the linear combination. However, note that the last layers of a CNN are usually fully-connected layers (and not convolutional or pooling layers), that is, they are a linear combination of their input and their weights followed by an application of an activation function.
  
  > **Comments:**
  > **username_1**: **Advice**: _you should expect different, ambiguous, inconsistent and even contradictory notations and terminologies that are used to describe the same (or similar) concepts_. You need to be able to deal with these unfortunate inconsistencies.  
  
  ---
  
  