Figure 3: Mean zero-shot accuracy over Wino-
grande, HellaSwag, PiQA, Arc-Easy, and Arc-
Challenge using LLaMA models with different 4-bit
data types. The NormalFloat data type significantly
improves the bit-for-bit accuracy gains compared
to regular 4-bit Floats. While Double Quantization
(DQ) only leads to minor gains, it allows for a more
fine-grained control over the memory footprint to fit
models of certain size (33B/65B) into certain GPUs
(24/48GB).

Figure 2: RougeL for LLaMA 7B models on the
Alpaca dataset. Each point represents a run with a
different random seed. We improve on the Stanford
Alpaca fully finetuned default hyperparameters to
construct a strong 16-bit baseline for comparisons.
Using LoRA on all transformer layers is critical to
match 16-bit performance.

6
QLORA