Answers.Body: <p>The regularization effect of Batch Normalization (BN) is a <em>side effect</em>, since it introduces noise during training because it uses batch statistics (first and second moments) to normalize the data.</p>
  <p>I guess that, in the implementation that you have seen, BN is used as the last layer just to provide an output that approximately follows a Gaussian distribution (that can be 0-mean and unitary variance, for example), and so on for the purpose of regularization.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-14T04:32:20.203Z
Answers.Text: 1. Why does it cause the output to approximately follow a Gaussian distribution?
Answers.UserDisplayName: Richie Bendall
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-14T04:32:37.993Z
Answers.Text: 2. Why is it useful for the output to approximately follow a Gaussian distribution?
Answers.UserDisplayName: Richie Bendall
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-14T07:30:08.917Z
Answers.Text: 1) Because BN standardizes the batch, by subtracting mean and dividing by variance. So it's like transforming the data to be Gaussian-like. Then, it scales and shifts them to provide more flexibility as a 0-mean 1-variance normal can be too restrictive. 2) It can be due to the data be normalized in the same way, but also to ensure better convergence given a specific loss function, or even to comply to some theoretical property of the problem (but that is beyond Inception I guess.)
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-13T19:33:42.067Z
Answers.LastActivityDate: 2023-04-15T07:16:28.680Z
Answers.LastEditDate: 2023-04-15T07:16:28.680Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>In <a href="https://github.com/timesler/facenet-pytorch/blob/fa70227bd5f02209512f60bd10e7e66877fdb4f6/models/inception_resnet_v1.py#L258" rel="nofollow noreferrer">this community implementation</a> of <a href="https://arxiv.org/pdf/1602.07261.pdf" rel="nofollow noreferrer">Inception-ResNet</a>, <a href="http://proceedings.mlr.press/v37/ioffe15.pdf" rel="nofollow noreferrer">Batch Normalization</a> is used as the final layer.</p>
  <p>This is unlikely to be a mistake, but also seems counterintuitive. The <a href="https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Szegedy_Rethinking_the_Inception_CVPR_2016_paper.pdf" rel="nofollow noreferrer">inceptionv2 paper</a> suggests that Batch Normalization acts as a regularizer. But wouldn't it be too strong of a regularizer as the very last layer?</p>
  
Tags: <batch-normalization><network-design>
Title: Batch Normalization as the Final Layer
UserDisplayName: Richie Bendall
ThreadText: # Batch Normalization as the Final Layer
  
  (Asked by: username_0 on 2023-04-13)
  
  In [this community implementation](https://github.com/timesler/facenet-pytorch/blob/fa70227bd5f02209512f60bd10e7e66877fdb4f6/models/inception_resnet_v1.py#L258) of [Inception-ResNet](https://arxiv.org/pdf/1602.07261.pdf), [Batch Normalization](http://proceedings.mlr.press/v37/ioffe15.pdf) is used as the final layer.
  
  This is unlikely to be a mistake, but also seems counterintuitive. The [inceptionv2 paper](https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Szegedy_Rethinking_the_Inception_CVPR_2016_paper.pdf) suggests that Batch Normalization acts as a regularizer. But wouldn't it be too strong of a regularizer as the very last layer?
  
  **Tags:** batch-normalization, network-design
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-13. Score: 0 | Accepted answer
  
  The regularization effect of Batch Normalization (BN) is a *side effect*, since it introduces noise during training because it uses batch statistics (first and second moments) to normalize the data.
  
  I guess that, in the implementation that you have seen, BN is used as the last layer just to provide an output that approximately follows a Gaussian distribution (that can be 0-mean and unitary variance, for example), and so on for the purpose of regularization.
  
  > **Comments:**
  > **username_0**: 1. Why does it cause the output to approximately follow a Gaussian distribution?  
  > **username_0**: 2. Why is it useful for the output to approximately follow a Gaussian distribution?  
  > **username_1**: 1) Because BN standardizes the batch, by subtracting mean and dividing by variance. So it's like transforming the data to be Gaussian-like. Then, it scales and shifts them to provide more flexibility as a 0-mean 1-variance normal can be too restrictive. 2) It can be due to the data be normalized in the same way, but also to ensure better convergence given a specific loss function, or even to comply to some theoretical property of the problem (but that is beyond Inception I guess.)  
  
  ---
  
  