not provide hard measurements for Paged Optimiz-
ers since the paging only occurs when processing
mini-batches with long sequence lengths, which is
rare. We do, however, perform an analysis of the
runtime of paged optimizers for 65B models on
48GB GPUs and find that with a batch size of 16,
paged optimizers provide the same training speed
as regular optimizers. Future work should measure
and characterize under what circumstances slow-
downs occur from the paging process.
Default LoRA hyperparameters do not match 16-
bit performance When using the standard prac-
tice of applying LoRA to query and value attention
projection matrices [28], we are not able to replicate
full finetuning performance for large base models.
As shown in Figure 2 for LLaMA 7B finetuning on
Alpaca, we find that the most critical LoRA hyper-
parameter is how many LoRA adapters are used in
total and that LoRA on all linear transformer block
layers are required to match full finetuning perfor-
QLORA