Y = XW + sXL1L2,

(3)

where L1 ∈ Rh×r and L2 ∈ Rr×o, and s is a scalar.
Memory Requirement of Parameter-Efficient Finetuning One important point of discussion is
the memory requirement of LoRA during training both in terms of the number and size of adapters
used. Since the memory footprint of LoRA is so minimal, we can use more adapters to improve
performance without significantly increasing the total memory used. While LoRA was designed as a

(1)

(2)

3
QLORA