Since pretrained neural network weights usually have a zero-centered normal distribution with
standard deviation σ (see Appendix F), we can transform all weights to a single fixed distribution by
scaling σ such that the distribution fits exactly into the range of our data type. For our data type, we
set the arbitrary range [−1, 1]. As such, both the quantiles for the data type and the neural network
weights need to be normalized into this range.
The information theoretically optimal data type for zero-mean normal distributions with arbitrary
standard deviations σ in the range [−1, 1] is computed as follows: (1) estimate the 2k + 1 quantiles
of a theoretical N (0, 1) distribution to obtain a k-bit quantile quantization data type for normal distri-
butions, (2) take this data type and normalize its values into the [−1, 1] range, (3) quantize an input
weight tensor by normalizing it into the [−1, 1] range through absolute maximum rescaling.
QLORA