Machine learning technique
 ;templatestyles src="Machine learning/styles.css"/ ;
In machine learning, normalization is a statistical technique with various applications. There are two main forms of normalization, namely "data normalization" and "activation normalization". Data normalization (or feature scaling) includes methods that rescale input data so that the features have the same range, mean, variance, or other statistical properties. For instance, a popular choice of feature scaling method is min-max normalization, where each feature is transformed to have the same range (typically formula_1 or formula_2). This solves the problem of different features having vastly different scales, for example if one feature is measured in kilometers and another in nanometers.
Activation normalization, on the other hand, is specific to deep learning, and includes methods that rescale the activation of hidden neurons inside neural networks.
Normalization is often used to:
Normalization techniques are often theoretically justified as reducing covariance shift, smoothing optimization landscapes, and increasing regularization, though they are mainly justified by empirical success.
Batch normalization.
Batch normalization (BatchNorm) operates on the activations of a layer for each mini-batch.
Consider a simple feedforward network, defined by chaining together modules:
formula_3
where each network module can be a linear transform, a nonlinear activation function, a convolution, etc. formula_4 is the input vector, formula_5 is the output vector from the first module, etc.
BatchNorm is a module that can be inserted at any point in the feedforward network. For example, suppose it is inserted just after formula_6, then the network would operate accordingly:
formula_7
The BatchNorm module does not operate over individual inputs. Instead, it must operate over one batch of inputs at a time.
Concretely, suppose we have a batch of inputs formula_8, fed all at once into the network. We would obtain in the middle of the network some vectors:
formula_9
The BatchNorm module computes the coordinate-wise mean and variance of these vectors:
formula_10
where formula_11 indexes the coordinates of the vectors, and formula_12 indexes the elements of the batch. In other words, we are considering the formula_11-th coordinate of each vector in the batch, and computing the mean and variance of these numbers.
It then normalizes each coordinate to have zero mean and unit variance:
formula_14
The formula_15 is a small positive constant such as formula_16 added to the variance for numerical stability, to avoid division by zero.
Finally, it applies a linear transformation:
formula_17
Here, formula_18 and formula_19 are parameters inside the BatchNorm module. They are learnable parameters, typically trained by gradient descent.
The following is a Python implementation of BatchNorm:
import numpy as np
def batchnorm(x, gamma, beta, epsilon=1e-9):
 # Mean and variance of each feature
 mu = np.mean(x, axis=0) # shape (N,)
 var = np.var(x, axis=0) # shape (N,)
 # Normalize the activations
 x_hat = (x - mu) / np.sqrt(var + epsilon) # shape (B, N)
 # Apply the linear transform
 y = gamma * x_hat + beta # shape (B, N)
 return y
Interpretation.
formula_18 and formula_19 allow the network to learn to undo the normalization, if this is beneficial. BatchNorm can be interpreted as removing the purely linear transformations, so that its layers focus solely on modelling the nonlinear aspects of data, which may be beneficial, as a neural network can always be augmented with a linear transformation layer on top.
It is claimed in the original publication that BatchNorm works by reducing internal covariance shift, though the claim has both supporters and detractors.
Special cases.
The original paper recommended to only use BatchNorms after a linear transform, not after a nonlinear activation. That is, formula_22, not formula_23. Also, the bias formula_12 does not matter, since it would be canceled by the subsequent mean subtraction, so it is of the form formula_25. That is, if a BatchNorm is preceded by a linear transform, then that linear transform's bias term is set to zero.
For convolutional neural networks (CNNs), BatchNorm must preserve the translation-invariance of these models, meaning that it must treat all outputs of the same kernel as if they are different data points within a batch. This is sometimes called Spatial BatchNorm, or BatchNorm2D, or per-channel BatchNorm.
Concretely, suppose we have a 2-dimensional convolutional layer defined by:
formula_26
where:
In order to preserve the translational invariance, BatchNorm treats all outputs from the same kernel in the same batch as more data in a batch. That is, it is applied once per "kernel" formula_29 (equivalently, once per channel formula_29), not per "activation" formula_40:
formula_41
where formula_42 is the batch size, formula_43 is the height of the feature map, and formula_44 is the width of the feature map.
That is, even though there are only formula_42 data points in a batch, all formula_46 outputs from the kernel in this batch are treated equally.
Subsequently, normalization and the linear transform is also done per kernel:
formula_47
Similar considerations apply for BatchNorm for "n"-dimensional convolutions.
The following is a Python implementation of BatchNorm for 2D convolutions:
import numpy as np
def batchnorm_cnn(x, gamma, beta, epsilon=1e-9):
 # Calculate the mean and variance for each channel.
 mean = np.mean(x, axis=(0, 1, 2), keepdims=True)
 var = np.var(x, axis=(0, 1, 2), keepdims=True)
 # Normalize the input tensor.
 x_hat = (x - mean) / np.sqrt(var + epsilon)
 # Scale and shift the normalized tensor.
 y = gamma * x_hat + beta
 return y
Improvements.
BatchNorm has been very popular and there were many attempted improvements. Some examples include:
A particular problem with BatchNorm is that during training, the mean and variance are calculated on the fly for each batch (usually as an exponential moving average), but during inference, the mean and variance were frozen from those calculated during training. This train-test disparity degrades performance. The disparity can be decreased by simulating the moving average during inference:Eq. 3
formula_50
where formula_51 is a hyperparameter to be optimized on a validation set.
Other works attempt to eliminate BatchNorm, such as the Normalizer-Free ResNet.
Layer normalization.
Layer normalization (LayerNorm) is a popular alternative to BatchNorm. Unlike BatchNorm, which normalizes activations across the batch dimension for a given feature, LayerNorm normalizes across all the features within a single data sample. Compared to BatchNorm, LayerNorm's performance is not affected by batch size. It is a key component of transformer models.
For a given data input and layer, LayerNorm computes the mean formula_52 and variance formula_53 over all the neurons in the layer. Similar to BatchNorm, learnable parameters formula_18 (scale) and formula_19 (shift) are applied. It is defined by:
formula_56
where:
formula_57
and the index formula_11 ranges over the neurons in that layer.
Examples.
For example, in CNN, a LayerNorm applies to all activations in a layer. In the previous notation, we have:
formula_59
Notice that the batch index formula_12 is removed, while the channel index formula_29 is added.
In recurrent neural networks and transformers, LayerNorm is applied individually to each timestep. For example, if the hidden vector in an RNN at timestep formula_62 is formula_63, where formula_64 is the dimension of the hidden vector, then LayerNorm will be applied with:
formula_65
where:
formula_66
Root mean square layer normalization.
Root mean square layer normalization (RMSNorm) changes LayerNorm by:
formula_67
Essentially, it is LayerNorm where we enforce formula_68.
Adaptive.
Adaptive layer norm (adaLN) computes the formula_69 in a LayerNorm not from the layer activation itself, but from other data. It was first proposed for CNNs, and has been used effectively in diffusion transformers (DiTs). For example, in a DiT, the conditioning information (such as a text encoding vector) is processed by a multilayer perceptron into formula_69, which is then applied in the LayerNorm module of a transformer.
Weight normalization.
Weight normalization (WeightNorm) is a technique inspired by BatchNorm that normalizes weight matrices in a neural network, rather than its activations.
One example is spectral normalization, which divides weight matrices by their spectral norm. The spectral normalization is used in generative adversarial networks (GANs) such as the Wasserstein GAN. The spectral radius can be efficiently computed by the following algorithm:
 ;templatestyles src="Template:Blockquote/styles.css" / ;INPUT matrix formula_44 and initial guess formula_72
Iterate formula_73.
RETURN formula_74 after each update of the discriminator, we can upper-bound formula_75, and thus upper-bound formula_76.
The algorithm can be further accelerated by memoization: at step formula_62, store formula_78. Then, at step formula_79, use formula_78 as the initial guess for the algorithm. Since formula_81 is very close to formula_82, so is formula_78 to formula_84, thus allowing rapid convergence.
CNN-specific normalization.
There are some activation normalization techniques that are only used for CNNs.
Response normalization.
Local response normalization was used in AlexNet. It was applied in a convolutional layer, just after a nonlinear activation function. It was defined by:
formula_85
where formula_86 is the activation of the neuron at location formula_87 and channel formula_11. I.e., each pixel in a channel is suppressed by the activations of the same pixel in its adjacent channels.
formula_89 are hyperparameters picked by using a validation set.
It was a variant of the earlier local contrast normalization.
formula_90
where formula_91 is the average activation in a small window centered on location formula_87 and channel formula_11. The hyperparameters formula_89, and the size of the small window, are picked by using a validation set.
Similar methods were called divisive normalization, as they divide activations by a number depending on the activations. They were originally inspired by biology, where it was used to explain nonlinear responses of cortical neurons and nonlinear masking in visual perception.
Both kinds of local normalization were obviated by batch normalization, which is a more global form of normalization.
Response normalization reappeared in ConvNeXT-2 as global response normalization.
Group normalization.
Group normalization (GroupNorm) is a technique also solely used for CNNs. It can be understood as the LayerNorm for CNN applied once per channel group.
Suppose at a layer formula_30, there are channels formula_96, then it is partitioned into groups formula_97. Then, LayerNorm is applied to each group.
Instance normalization.
Instance normalization (InstanceNorm), or contrast normalization, is a technique first developed for neural style transfer, and is also only used for CNNs. It can be understood as the LayerNorm for CNN applied once per channel, or equivalently, as group normalization where each group consists of a single channel:
formula_98
Adaptive instance normalization.
Adaptive instance normalization (AdaIN) is a variant of instance normalization, designed specifically for neural style transfer with CNNs, rather than just CNNs in general.
In the AdaIN method of style transfer, we take a CNN and two input images, one for content and one for style. Each image is processed through the same CNN, and at a certain layer formula_30, AdaIn is applied.
Let formula_100 be the activation in the content image, and formula_101 be the activation in the style image. Then, AdaIn first computes the mean and variance of the activations of the content image formula_102, then uses those as the formula_69 for InstanceNorm on formula_100. Note that formula_101 itself remains unchanged. Explicitly, we have:
formula_106
Transformers.
Some normalization methods were designed for use in transformers.
The original 2017 transformer used the "post-LN" configuration for its LayerNorms. It was difficult to train, and required careful hyperparameter tuning and a "warm-up" in learning rate, where it starts small and gradually increases. The pre-LN convention, proposed several times in 2018, was found to be easier to train, requiring no warm-up, leading to faster convergence.
FixNorm and ScaleNorm both normalize activation vectors in a transformer. The FixNorm method divides the "output" vectors from a transformer by their L2 norms, then multiplies by a learned parameter formula_107. The ScaleNorm replaces all LayerNorms inside a transformer by division with L2 norm, then multiplying by a learned parameter formula_108 (shared by all ScaleNorm modules of a transformer). Query-Key normalization (QKNorm) normalizes query and key vectors to have unit L2 norm.
In nGPT, many vectors are normalized to have unit L2 norm: hidden state vectors, input and output embedding vectors, weight matrix columns, and query and key vectors.
Miscellaneous.
Gradient normalization (GradNorm) normalizes gradient vectors during backpropagation.