Answers.Body: <p>You don't put batch normalization or dropout layers after the last layer, it will just &quot;corrupt&quot; your predictions. They are intended to be used only within the network, to help it converge and avoid overfitting.</p>
  <p>BTW even if your fully connected layer's output is always positive, it would have positive and negative outputs after batch normalization. But as I said you shouldn't have that layer there anyway.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-18T15:33:24.743Z
Answers.LastActivityDate: 2022-01-18T15:33:24.743Z
Answers.UserDisplayName: NikoNyrh
Answers.Body: <p>No, the activation of the output layer should instead be tailored to the labels you're trying to predict. The network prediction can be seen as a distribution, for example a categorical for classification or a Gaussian (or something more flexible) for regression. The output of your network should predict the sufficient statistics of this distribution. For example, a softmax activation on the last layer ensures that the outputs are positive and sum up to one, as you would expect for a categorical distribution. When you predict a Gaussian with mean and variance, you don't need an activation for the mean but the variance has to be positive, so you could use exp as activation for that part of the output.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-01T13:05:45.543Z
Answers.Text: Okay, and what about dropout/batch-norm? Should they be after the last fully connected layer?
Answers.UserDisplayName: Gilad Deutsch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-28T04:05:37.513Z
Answers.LastActivityDate: 2020-04-28T04:05:37.513Z
Answers.UserDisplayName: danijar
Answers.Body: <p>There are both benefits and drawbacks to using batch normalization , relu &amp; dropout after the last fully connected layer.</p>
  <p>One benefit of using this combination is that it can help to prevent overfitting, as the batch normalization and dropout provide additional regularization.</p>
  <p>However, using this combination may also make the model more difficult to train, as the batch normalization <em>can introduce additional vanishing gradients</em>. Ultimately, it is up to the practitioner to decide whether to use this combination after the last fully connected layer, based on the specific model and dataset.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-17T13:03:19.273Z
Answers.LastActivityDate: 2022-10-17T13:03:19.273Z
Answers.UserDisplayName: Faizy
Answers.Body: <p>If your output is always positive (say from zero to infinity), I guess it won't hurt to put a relu after the last layer.</p>
  <p>Note that in the case where your output is probabilities (from range 0 to 1, say for classification), people generally apply a sigmoid or softmax (depending whether the task is mutilabel or not) after the last layer. But this is equivalent to not applying any activation to the output, and instead interpret the output as logits (say setting the logits=True flag true in the loss function). Choosing between the two is largely a matter of software-engineering style.</p>
  <p>As for the question of whether to apply batchnorm after the last layer, it doesn't make much sense to do so, because in which case each output node is always of mean zero, so it's hard to make any strong prediction (even if you were to apply a relu after it). Actually, I think most people do not even use batchnorm <strong>before</strong> the last layer, but the reason for this is more empirical that theoretically justified.</p>
  <p>And you definitely do not want to apply dropout after the last layer, which would result in the correct prediction being occasionally dropped..</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-17T06:01:07.147Z
Answers.LastActivityDate: 2022-10-17T06:01:07.147Z
Answers.UserDisplayName: PeaBrane
Body: <p>I am using the following architechture:</p>
  
  <pre><code>3*(fully connected -&gt; batch normalization -&gt; relu -&gt; dropout) -&gt; fully connected
  </code></pre>
  
  <p>Should I add the <code>batch normalization -&gt; relu -&gt; dropout</code> part after the last fully connected layer as well (the output is positive anyway, so the relu wouldn't hurt I suppose)? </p>
  
Tags: <neural-networks><deep-learning><architecture>
Title: Should batch-normalization/dropout/activation-function layers be used after the last fully connected layer?
UserDisplayName: Gilad Deutsch
ThreadText: # Should batch-normalization/dropout/activation-function layers be used after the last fully connected layer?
  
  (Asked by: username_0 on 2020-04-27)
  
  I am using the following architechture:
  
  ```
  3*(fully connected -> batch normalization -> relu -> dropout) -> fully connected
  
  ```
  
  Should I add the `batch normalization -> relu -> dropout` part after the last fully connected layer as well (the output is positive anyway, so the relu wouldn't hurt I suppose)?
  
  **Tags:** neural-networks, deep-learning, architecture
  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2022-01-18. Score: 3
  
  You don't put batch normalization or dropout layers after the last layer, it will just "corrupt" your predictions. They are intended to be used only within the network, to help it converge and avoid overfitting.
  
  BTW even if your fully connected layer's output is always positive, it would have positive and negative outputs after batch normalization. But as I said you shouldn't have that layer there anyway.
  
  
  ---
  
  ### Answer by username_2 on 2020-04-28. Score: 0
  
  No, the activation of the output layer should instead be tailored to the labels you're trying to predict. The network prediction can be seen as a distribution, for example a categorical for classification or a Gaussian (or something more flexible) for regression. The output of your network should predict the sufficient statistics of this distribution. For example, a softmax activation on the last layer ensures that the outputs are positive and sum up to one, as you would expect for a categorical distribution. When you predict a Gaussian with mean and variance, you don't need an activation for the mean but the variance has to be positive, so you could use exp as activation for that part of the output.
  
  > **Comments:**
  > **username_0**: Okay, and what about dropout/batch-norm? Should they be after the last fully connected layer?  
  
  ---
  
  ### Answer by username_3 on 2022-10-17. Score: 0
  
  There are both benefits and drawbacks to using batch normalization , relu & dropout after the last fully connected layer.
  
  One benefit of using this combination is that it can help to prevent overfitting, as the batch normalization and dropout provide additional regularization.
  
  However, using this combination may also make the model more difficult to train, as the batch normalization *can introduce additional vanishing gradients*. Ultimately, it is up to the practitioner to decide whether to use this combination after the last fully connected layer, based on the specific model and dataset.
  
  
  ---
  
  ### Answer by username_4 on 2022-10-17. Score: 0
  
  If your output is always positive (say from zero to infinity), I guess it won't hurt to put a relu after the last layer.
  
  Note that in the case where your output is probabilities (from range 0 to 1, say for classification), people generally apply a sigmoid or softmax (depending whether the task is mutilabel or not) after the last layer. But this is equivalent to not applying any activation to the output, and instead interpret the output as logits (say setting the logits=True flag true in the loss function). Choosing between the two is largely a matter of software-engineering style.
  
  As for the question of whether to apply batchnorm after the last layer, it doesn't make much sense to do so, because in which case each output node is always of mean zero, so it's hard to make any strong prediction (even if you were to apply a relu after it). Actually, I think most people do not even use batchnorm **before** the last layer, but the reason for this is more empirical that theoretically justified.
  
  And you definitely do not want to apply dropout after the last layer, which would result in the correct prediction being occasionally dropped..
  
  
  ---
  
  