Answers.Body: <p>Batch-normalization (BN) does NOT <em>transform all values by restricting them to a value between zero and one.</em></p>
  <p>BN performs two operations: a <em>normalization</em>, and a <em>shifting with scaling</em>. The normalization operation transforms the inputs to have approximately zero-mean and unitary variance, then the scaling by <span class="math-container">$\gamma$</span> and shift by <span class="math-container">$\beta$</span> is performed to give more power (or flexibility) to the next layers.</p>
  <p>In general you want to apply BN before any activation function (like ReLU) because the BN tend to center the data in the &quot;active&quot; region of the non-linearity, thus speeding-up training.</p>
  <p>Moreover, BN introduces some noise in the training which is often interpreted as an implicit form of regularization (thus boosting also generalization performance.)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-11T23:01:09.330Z
Answers.Text: If BN is applied before ReLU, couldn't be the case that all inputs to BN are positive, so after passing through the BN layer some of the become negative leading to dead neurons?
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-12T08:14:40.920Z
Answers.Text: I guess this possibility cannot be excluded with certainty, but consider that BN also learns a shifting coefficient $\beta$ that is added after standardization. So the learning dynamics may push $\beta$ to be large enough to still yield positive values that would be active after the ReLU.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-11T13:29:23.440Z
Answers.LastActivityDate: 2023-04-11T13:29:23.440Z
Answers.UserDisplayName: Luca Anzalone
Answers.Body: <p>Although you can normalize the features and initialize the weights with Normal Xavier initialization, during propagation the weights size can still propagate through a deep neural network, causing Vanishing/exploding gradients. Think about &quot;very deep&quot; neural networks with +50 layers, how easily the weights could go bananas.</p>
  <p>Batch-normalization normalizes after each batch, continuously keeping the weights in check.</p>
  <p>You also write it &quot;works&quot; but that does not say anything about performance. It might be better to use Leaky ReLU to avoid dead neurons which seem very possible with the a feature and weight normalized relu activation neural network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-11T14:12:29.993Z
Answers.LastActivityDate: 2023-04-11T14:12:29.993Z
Answers.UserDisplayName: nammerkage
Body: <p>I have read a little about Batch-Normalization and I understood that there isn't any better option on where you place Batch-Normalization (it all depends on the case).
  However, I don't understand the point on applying Batch-Normalization followed by a ReLU.</p>
  <p>ReLU transform all negative values to zero. But Batch-Normalization already transforms all values by restricting them to a value between zero and one. So ReLU will not have any effect when applied after BN, because there will not be any negative values left.</p>
  <p>Am I right, or am I missing something?</p>
  
Tags: <batch-normalization><relu>
Title: Why does Batch-Normalization before ReLU work?
UserDisplayName: Bliscor
ThreadText: # Why does Batch-Normalization before ReLU work?
  
  (Asked by: username_0 on 2023-04-10)
  
  I have read a little about Batch-Normalization and I understood that there isn't any better option on where you place Batch-Normalization (it all depends on the case).
  However, I don't understand the point on applying Batch-Normalization followed by a ReLU.
  
  ReLU transform all negative values to zero. But Batch-Normalization already transforms all values by restricting them to a value between zero and one. So ReLU will not have any effect when applied after BN, because there will not be any negative values left.
  
  Am I right, or am I missing something?
  
  **Tags:** batch-normalization, relu
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-04-11. Score: 2
  
  Batch-normalization (BN) does NOT *transform all values by restricting them to a value between zero and one.*
  
  BN performs two operations: a *normalization*, and a *shifting with scaling*. The normalization operation transforms the inputs to have approximately zero-mean and unitary variance, then the scaling by $\gamma$ and shift by $\beta$ is performed to give more power (or flexibility) to the next layers.
  
  In general you want to apply BN before any activation function (like ReLU) because the BN tend to center the data in the "active" region of the non-linearity, thus speeding-up training.
  
  Moreover, BN introduces some noise in the training which is often interpreted as an implicit form of regularization (thus boosting also generalization performance.)
  
  > **Comments:**
  > **username_2**: If BN is applied before ReLU, couldn't be the case that all inputs to BN are positive, so after passing through the BN layer some of the become negative leading to dead neurons?  
  > **username_1**: I guess this possibility cannot be excluded with certainty, but consider that BN also learns a shifting coefficient $\beta$ that is added after standardization. So the learning dynamics may push $\beta$ to be large enough to still yield positive values that would be active after the ReLU.  
  
  ---
  
  ### Answer by username_3 on 2023-04-11. Score: 0
  
  Although you can normalize the features and initialize the weights with Normal Xavier initialization, during propagation the weights size can still propagate through a deep neural network, causing Vanishing/exploding gradients. Think about "very deep" neural networks with +50 layers, how easily the weights could go bananas.
  
  Batch-normalization normalizes after each batch, continuously keeping the weights in check.
  
  You also write it "works" but that does not say anything about performance. It might be better to use Leaky ReLU to avoid dead neurons which seem very possible with the a feature and weight normalized relu activation neural network.
  
  
  ---
  
  