Answers.Body: <p>Layer normalization normalizes a batch of data <span class="math-container">$X$</span> (usually the output of some intermediate hidden layer) with dimensionality <span class="math-container">$D$</span> (e.g. <span class="math-container">$D$</span> features/columns), independently on each dimension considering the statistics (e.g. mean and variance) computed on the current batch.</p>
  <p>Consider that your batch comprises randomly picked data points, so if your batch size is too small the sampled data won't be representative of the whole dataset.</p>
  <p>But even in such situation your classifier is not said to fail since it mostly (if not entirely) relies on (higher-order) correlations among the features and class label.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-23T20:33:22.260Z
Answers.Text: Thanks Luca. 
  I understand from your explanation (layernorm applies on output of hidden layer) that any mapping happening before the layernorm could take care of the time (raw) features in the input.
Answers.UserDisplayName: Meisam Ashraf
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-23T20:48:26.083Z
Answers.Text: Exactly, usually LN is applied after hidden layers. To normalize the input, if necessary, one can use batch-normalization (which keeps updating a moving average of mean and std statistics.)
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-22T18:54:40.437Z
Answers.LastActivityDate: 2023-05-22T18:54:40.437Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>When applying layer normalization in a neural network, we consider the mean and variance of each input data point to the layer separately without considering the global statistics in the entire data-set. How does the model learn to, for example, discriminate for a classification based on global statistics?</p>
  <p>If we assume a classifier for input 1D time signals of sine waves with a fixed frequency. If we label the signals by amplitude for classification (say high and low amplitude in a binary classifier), aren't we losing the amplitude information in the layernorm?</p>
  <pre><code>import torch
  import holoviews as hv
  
  hv.extension(&quot;bokeh&quot;)
  
  TWOPI = 2 * torch.pi
  N = 1000
  fr = 10
  ratio = 5
  
  t = torch.linspace(0, 1, N)
  x1 = torch.sin(TWOPI * fr * t)
  x2 = ratio * x1
  
  hx1 = hv.Curve((t, x1.numpy())).opts(xlabel=&quot;Time&quot;, ylabel=&quot;Amplitude&quot;, color=&quot;black&quot;)
  hx2 = hv.Curve((t, x2.numpy())).opts(
      xlabel=&quot;Time&quot;, ylabel=&quot;Amplitude&quot;, color=&quot;red&quot;, line_dash=&quot;dashed&quot;
  )
  
  
  lyn = torch.nn.LayerNorm(N)
  
  y1 = lyn(x1)
  y2 = lyn(x2)
  
  hy1 = hv.Curve((t, y1.detach().numpy())).opts(
      xlabel=&quot;Time&quot;, ylabel=&quot;Normalized Amplitude&quot;, color=&quot;black&quot;
  )
  hy2 = hv.Curve((t, y2.detach().numpy())).opts(
      xlabel=&quot;Time&quot;, ylabel=&quot;Normalized Amplitude&quot;, color=&quot;red&quot;, line_dash=&quot;dashed&quot;
  )
  
  hx1 * hx2 + hy1 * hy2
  </code></pre>
  <p><a href="https://i.sstatic.net/aZ2wo.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/aZ2wo.png" alt="enter image description here" /></a></p>
  
Tags: <neural-networks><normalisation>
Title: Can we model the global statistics of the dataset features using LayerNorm?
UserDisplayName: Meisam Ashraf
ThreadText: # Can we model the global statistics of the dataset features using LayerNorm?
  
  (Asked by: username_0 on 2023-05-22)
  
  When applying layer normalization in a neural network, we consider the mean and variance of each input data point to the layer separately without considering the global statistics in the entire data-set. How does the model learn to, for example, discriminate for a classification based on global statistics?
  If we assume a classifier for input 1D time signals of sine waves with a fixed frequency. If we label the signals by amplitude for classification (say high and low amplitude in a binary classifier), aren't we losing the amplitude information in the layernorm?
  import torch
  import holoviews as hv
  
  hv.extension("bokeh")
  
  TWOPI = 2 * torch.pi
  N = 1000
  fr = 10
  ratio = 5
  
  t = torch.linspace(0, 1, N)
  x1 = torch.sin(TWOPI * fr * t)
  x2 = ratio * x1
  
  hx1 = hv.Curve((t, x1.numpy())).opts(xlabel="Time", ylabel="Amplitude", color="black")
  hx2 = hv.Curve((t, x2.numpy())).opts(
      xlabel="Time", ylabel="Amplitude", color="red", line_dash="dashed"
  )
  
  
  lyn = torch.nn.LayerNorm(N)
  
  y1 = lyn(x1)
  y2 = lyn(x2)
  
  hy1 = hv.Curve((t, y1.detach().numpy())).opts(
      xlabel="Time", ylabel="Normalized Amplitude", color="black"
  )
  hy2 = hv.Curve((t, y2.detach().numpy())).opts(
      xlabel="Time", ylabel="Normalized Amplitude", color="red", line_dash="dashed"
  )
  
  hx1 * hx2 + hy1 * hy2
  
  **Tags:** neural-networks, normalisation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-05-22. Score: 1
  
  Layer normalization normalizes a batch of data $X$ (usually the output of some intermediate hidden layer) with dimensionality $D$ (e.g. $D$ features/columns), independently on each dimension considering the statistics (e.g. mean and variance) computed on the current batch.
  
  Consider that your batch comprises randomly picked data points, so if your batch size is too small the sampled data won't be representative of the whole dataset.
  
  But even in such situation your classifier is not said to fail since it mostly (if not entirely) relies on (higher-order) correlations among the features and class label.
  
  > **Comments:**
  > **username_0**: Thanks Luca.  I understand from your explanation (layernorm applies on output of hidden layer) that any mapping happening before the layernorm could take care of the time (raw) features in the input.  
  > **username_1**: Exactly, usually LN is applied after hidden layers. To normalize the input, if necessary, one can use batch-normalization (which keeps updating a moving average of mean and std statistics.)  
  
  ---
  
  