∆W = WdowmWup
∆W = Wdown ⊗ Wup
∆W = Wdown↓bWup↓b

∆W = P ΛQ

∆W = WdownΛWup
∆W = WdownWup

∆W = WdownWup ⊙ M
down W BF 16
∆W = W BF 16
up
∆W = WdownWup
∆W = SVD(W − Qt)
Ah

(cid:32)

(cid:33)

∆W h = (BLoRA, Bh)

LoRA
Ah

Notes
F : Rr → Rd, W r ∈ Rr is parameters to be optimized, and r ≪ d.
Wdown ∈ Rk×r, Wup ∈ Rr×d, and r ≪ {k, d}.
rank(Wdown ⊗ Wup) = rank(Wdown) × rank(Wup).
Wdown↓b = Wdown[: b, :], Wup↓b = Wup[:, : b], b ∈ {rmin,··· , rmax}.
P P T = P T P = I = QQT = QT Q, Λ = diag(σ1, σ2, . . . , σr).
Λ = [λ1, λ2,··· , λr] with λi could be an arbitrary constant.
W (t+1) ← W (t) + (W (t+1)
δ = (W + WdownWup) ⊙ M, M ∈ {0, 1}1×G, G is group number.
Y BF 16 = X BF 16doubleDequant(cF P 32
Wdown ∈ Rk×r, Wup ∈ Rr×L, L is the quantization group number of W .
Qt = qN (W − W t−1
BLoRA is shared across all heads, Bh, Ah provide rank-r update in each head.

up ), qN is N-bit quantization function.

up − W (t)

, W N F 4) + X BF 16W BF 16

downW t−1

down W (t+1)

downW (t)
up ).
Parameter-EfficientFine-TuningMethods