Answers.Body: <p>Here's some examples:</p>
  <ul>
  <li><a href="https://arxiv.org/pdf/1803.08494.pdf" rel="nofollow noreferrer">Group Normalization</a></li>
  <li><a href="https://arxiv.org/pdf/1607.06450.pdf" rel="nofollow noreferrer">Layer Normalization</a></li>
  <li><a href="https://arxiv.org/pdf/1907.10473.pdf" rel="nofollow noreferrer">Switchable Normalization</a></li>
  <li><a href="https://arxiv.org/pdf/1908.01259.pdf" rel="nofollow noreferrer">Attentive Normalization</a></li>
  <li><a href="https://arxiv.org/pdf/1802.05957.pdf" rel="nofollow noreferrer">Spectrl Normalization</a></li>
  </ul>
  <p>Notice how in general different normalization techniques are implemented using as a backbone the same idea of batch normalization, i.e. computing mean and standard deviation, but along a different dimension rather than the batch one, or on a subset of points (image below taken from the switchable normalization paper)</p>
  <p><a href="https://i.sstatic.net/6l0Dx.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6l0Dx.png" alt="enter image description here" /></a></p>
  <p>The alternative, as performed by the attentive and spectral normalization authors, is to rethink the normalization as something different from the classic linear transformation <span class="math-container">$\mu x + \sigma$</span>, or to add extra parameters to learn, to make the normalization more non linear (image taken from the attentive normalization paper).</p>
  <p><a href="https://i.sstatic.net/kuuZZ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/kuuZZ.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-15T14:17:01.323Z
Answers.LastActivityDate: 2022-02-15T14:17:01.323Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>Consider the following paragraph, taken from <strong>OPTIMIZING BATCHES</strong> of the textbook named <strong><a href="https://web.archive.org/web/20220113022756/https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf" rel="nofollow noreferrer">Deep Learning with PyTorch</a></strong> by <em>Eli Stevens et al.</em>, regarding the reasons for processing data into batches</p>
  <blockquote>
  <p>The reason we want to do this batching is multifaceted. One big
  motivation is to make sure the computation we’re asking for is big
  enough to saturate the computing resources we’re using to perform the
  computation. GPUs in particular are highly parallelized, so a single
  input on a small model will leave most of the computing units idle. By
  providing batches of inputs, the calculation can be spread across the
  otherwise-idle units, which means the batched results come back just
  as quickly as a single result would. <strong>Another benefit is that some
  advanced models use statistical information from the entire batch, and
  those statistics get better with larger batch sizes.</strong></p>
  </blockquote>
  <p>The bolded portion says a statistical reason for batch processing. I know only batch normalization that satisfies the portion as mean and standard deviation are calculated over the batch of inputs.</p>
  <p>Is there any such example that needs statistics of batches that are apt to the description?</p>
  
Tags: <batch-normalization><statistics><batch-size>
Title: What can be an example other than batch normalization that uses statistics of batches?
UserDisplayName: hanugm
ThreadText: # What can be an example other than batch normalization that uses statistics of batches?
  
  (Asked by: username_0 on 2022-02-15)
  
  Consider the following paragraph, taken from **OPTIMIZING BATCHES** of the textbook named **[Deep Learning with PyTorch](https://web.archive.org/web/20220113022756/https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf)** by *Eli Stevens et al.*, regarding the reasons for processing data into batches
  
  > The reason we want to do this batching is multifaceted. One big
  > motivation is to make sure the computation we’re asking for is big
  > enough to saturate the computing resources we’re using to perform the
  > computation. GPUs in particular are highly parallelized, so a single
  > input on a small model will leave most of the computing units idle. By
  > providing batches of inputs, the calculation can be spread across the
  > otherwise-idle units, which means the batched results come back just
  > as quickly as a single result would. **Another benefit is that some
  > advanced models use statistical information from the entire batch, and
  > those statistics get better with larger batch sizes.**
  
  The bolded portion says a statistical reason for batch processing. I know only batch normalization that satisfies the portion as mean and standard deviation are calculated over the batch of inputs.
  
  Is there any such example that needs statistics of batches that are apt to the description?
  
  **Tags:** batch-normalization, statistics, batch-size
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-02-15. Score: 1
  
  Here's some examples:
  
  * [Group Normalization](https://arxiv.org/pdf/1803.08494.pdf)
  * [Layer Normalization](https://arxiv.org/pdf/1607.06450.pdf)
  * [Switchable Normalization](https://arxiv.org/pdf/1907.10473.pdf)
  * [Attentive Normalization](https://arxiv.org/pdf/1908.01259.pdf)
  * [Spectrl Normalization](https://arxiv.org/pdf/1802.05957.pdf)
  
  Notice how in general different normalization techniques are implemented using as a backbone the same idea of batch normalization, i.e. computing mean and standard deviation, but along a different dimension rather than the batch one, or on a subset of points (image below taken from the switchable normalization paper)
  
  [![enter image description here](https://i.sstatic.net/6l0Dx.png)](https://i.sstatic.net/6l0Dx.png)
  
  The alternative, as performed by the attentive and spectral normalization authors, is to rethink the normalization as something different from the classic linear transformation $\mu x + \sigma$, or to add extra parameters to learn, to make the normalization more non linear (image taken from the attentive normalization paper).
  
  [![enter image description here](https://i.sstatic.net/kuuZZ.png)](https://i.sstatic.net/kuuZZ.png)
  
  
  ---
  
  