Answers.Body: <p>Keep in mind that BN (batch-norm) is suggested with deep networks and large batch sizes (the larger, the better). Let's say that for a 2 or 3 layer neural net, BN may not play a big role so in most cases you can avoid that. Moreover, the larger the batch the more accurate (or less noisy) the normalization in BN will be thus improving convergence speed.</p>
  <p>In general, you use BN <em>after</em> each hidden layer and <em>before</em> the activation function - but seems that using BN after the activation works equally well. In principle, you can apply it also on top of the output layer if you expect the predictions (of a <em>regression</em> problem) to be Gaussian-like. If so BN standardizes them (i.e. approx zero-mean unit-var), also rescaling and reshifting them to provide flexibility.</p>
  <p>For example for models that yield an embedding of the inputs (e.g. AE, siamese nets, etc) you <em>never</em> apply BN on the embedding (which is usually l2-normalized.)</p>
  <p>Moreover, you can also apply BN on the input layer to avoid normalizing the inputs by yourself.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-13T08:08:17.543Z
Answers.LastActivityDate: 2023-06-13T08:08:17.543Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I am creating a neural network using <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html" rel="nofollow noreferrer">batchnorm</a> as a regularization method to enable deep models and prevent overfitting.</p>
  <p>I understand that <a href="https://medium.com/analytics-vidhya/internal-covariate-shift-an-overview-of-how-to-speed-up-neural-network-training-3e2a3dcdd5cc" rel="nofollow noreferrer">batchnorming supresses the internal covariance shift</a> introduced by randomly chosen minibatches, so it generally makes sense to add it to every layer in the model. However, the batchnorm also introduces <a href="https://towardsdatascience.com/batch-normalization-8a2e585775c9#:%7E:text=noise%20to%20your%20network" rel="nofollow noreferrer">noise into the training data</a>, which has a particularly heavy effect on model output in the last few layers of the model.</p>
  <p><strong>How exactly should I handle the batchnorm at the end of my model?</strong> It seems intuitive that <a href="https://stats.stackexchange.com/questions/361700/lack-of-batch-normalization-before-last-fully-connected-layer">the last layer should not be batchnormed</a>, as this would reduce the degrees of freedom for the output of the model and shift the outputs uncontrollably. But what about the penultimate layer and those before? Would it make sense to &quot;lighten&quot; the effects of the minibatch for example by reducing its impact gradually? Or is it better to just completely leave it out after some layer?</p>
  
Tags: <neural-networks><training><pytorch><regularization><batch-normalization>
Title: How to handle BatchNorm in the last layers of Neural Networks?
UserDisplayName: Quantum
ThreadText: # How to handle BatchNorm in the last layers of Neural Networks?
  
  (Asked by: username_0 on 2023-06-12)
  
  I am creating a neural network using [batchnorm](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html) as a regularization method to enable deep models and prevent overfitting.
  
  I understand that [batchnorming supresses the internal covariance shift](https://medium.com/analytics-vidhya/internal-covariate-shift-an-overview-of-how-to-speed-up-neural-network-training-3e2a3dcdd5cc) introduced by randomly chosen minibatches, so it generally makes sense to add it to every layer in the model. However, the batchnorm also introduces [noise into the training data](https://towardsdatascience.com/batch-normalization-8a2e585775c9#:%7E:text=noise%20to%20your%20network), which has a particularly heavy effect on model output in the last few layers of the model.
  
  **How exactly should I handle the batchnorm at the end of my model?** It seems intuitive that [the last layer should not be batchnormed](https://stats.stackexchange.com/questions/361700/lack-of-batch-normalization-before-last-fully-connected-layer), as this would reduce the degrees of freedom for the output of the model and shift the outputs uncontrollably. But what about the penultimate layer and those before? Would it make sense to "lighten" the effects of the minibatch for example by reducing its impact gradually? Or is it better to just completely leave it out after some layer?
  
  **Tags:** neural-networks, training, pytorch, regularization, batch-normalization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-06-13. Score: 1
  
  Keep in mind that BN (batch-norm) is suggested with deep networks and large batch sizes (the larger, the better). Let's say that for a 2 or 3 layer neural net, BN may not play a big role so in most cases you can avoid that. Moreover, the larger the batch the more accurate (or less noisy) the normalization in BN will be thus improving convergence speed.
  
  In general, you use BN *after* each hidden layer and *before* the activation function - but seems that using BN after the activation works equally well. In principle, you can apply it also on top of the output layer if you expect the predictions (of a *regression* problem) to be Gaussian-like. If so BN standardizes them (i.e. approx zero-mean unit-var), also rescaling and reshifting them to provide flexibility.
  
  For example for models that yield an embedding of the inputs (e.g. AE, siamese nets, etc) you *never* apply BN on the embedding (which is usually l2-normalized.)
  
  Moreover, you can also apply BN on the input layer to avoid normalizing the inputs by yourself.
  
  
  ---
  
  